Deploy detection algorithms and integrate results

This guide covers deployment, calibration, configuration, and result integration for MRDVS human, storage-location, and volume-detection algorithms. The algorithm runs on the camera; the algorithm host application is used for configuration, upgrades, and result inspection.

Algorithm firmware and the algorithm host application are delivered per project. Before flashing algorithm firmware, confirm that the camera is running an MDS firmware version. Obtain an installation package that matches the project and camera model.

Supported scope

System components

Component	Function
3D camera	Captures RGB images and point clouds and runs the detection algorithm on the camera.
Algorithm host application	Configures algorithms, calibrates extrinsics, displays results, and upgrades algorithm firmware.
Industrial PC or upper-layer system	Runs the host application or reads results through a network interface.
LiDAR	`mid360`, used for the original radar-based solution configuration.

Algorithms and cameras

Algorithm	Supported camera
Human detection	M4 Pro
Storage-location detection	S10 Pro
Volume detection	S10 and S10 Pro

For product information, see the S10 3D camera product page.

Host application environment

Item	Recommended configuration
Operating system	Windows 10 or later
CPU	Intel® Core™ i5 (7th generation) or later
Memory	8 GB or more
GPU	Intel® UHD Graphics or a higher-performance integrated or discrete GPU

Hardware and network deployment

Install the camera

Use a rigid bracket to prevent movement or vibration after installation. Human detection normally uses a horizontal installation or an installation with an angle of no more than 30°.

Connect the devices

Connect power and the network

Power the camera with a regulated 24 V DC supply. Use a Cat6A or Cat7 cable to connect the camera to a gigabit switch or controller.

Deploy multiple cameras

Install each camera at the height and position specified in the CAD plan. Connect all cameras through a gigabit switch and verify power and link status for each one.

Prepare test targets

Place the people, pallets, or goods to be detected in the camera view before calibrating extrinsics and configuring regions.

Configure the host network

Put the host computer and camera on the same subnet. The original example uses the following plan:

Item	Example
Host IP address	`192.168.100.xxx`
Subnet mask	`255.255.255.0`
Default gateway	`192.168.100.1`

Open adapter settings

In Windows, open Control Panel → Network and Sharing Center → Change adapter settings, then select the Ethernet adapter connected to the camera.

Configure IPv4

Open Internet Protocol Version 4 (TCP/IPv4), select Use the following IP address, and enter an unused static address on the camera’s subnet.

Confirm the connection

Save the settings, confirm a gigabit link, and use the host application to discover or open the camera.

Do not leave the firewall disabled on a production network. If the host application cannot connect, create a rule for the required application or port first. Temporarily disable the firewall only on a trusted debugging network, then restore protection.

Install and upgrade software

Install the host applications

Obtain the camera host application from the software downloads page.
Install the camera host application outside the system drive.
Copy the project-delivered algorithm host application files into the Tools folder under the camera host application directory. Replace files with the same name, then double-click the executable to start it.

To upgrade only the algorithm host application, replace its .exe file with the delivered version.

Upgrade algorithm firmware

The camera restarts during flashing. Do not disconnect power or the network or close the host application. The package must match the camera model, MDS firmware version, and project algorithm.

Camera host application
Algorithm host application

Start LxCameraViewer.exe and open the camera.
Select Basic tools → Function settings → Firmware upgrade.
Choose the delivered detection algorithm file and select Run.

When the camera appears again after reboot, flashing is complete. A green camera background in the algorithm host application means that the camera is connected and the algorithm is available.

Algorithm host application

Startup and status

Start LxApplicationViewer.exe. A green icon in the device list indicates a connected camera. Select a device to switch the current view between RGB and point-cloud modes. Standard users can view results; administrators can modify camera and algorithm parameters.

Basic tools

Function	Description
User switching	Switch permission levels. Administrators can configure camera and algorithm parameters.
Ideal ground	Show or hide the reference horizontal plane.
Extrinsic ground	Show or hide the ground calculated from extrinsic calibration.
Display cache	Leave unchanged.
Display algorithm	Show the currently enabled algorithm.
Export saved images	Export saved images.
Historical playback	View RGB images and ToF data logs.
Switch application algorithm	Choose the algorithm to run, then select Save and send.
Display detection results	Show the output from the current algorithm.

Status bar and playback

Function	Description
RGB	Show the RGB image.
Point cloud	Show the point cloud.
Stop algorithm	Stop the current algorithm. It does not restart automatically until the camera is powered on again.
Upgrade firmware	Open the algorithm firmware upgrade workflow.
Restart camera	Restart the selected camera.

When the status changes, the software saves one RGB frame and point-cloud data. Playback data is not shared between algorithms. Playback time uses the camera’s internal clock, not the host computer’s system time. If the device background is yellow, or it is green but the image does not update, enable automatic refresh to resume image acquisition.

Extrinsic calibration

Camera configuration

Configuration	Description
Camera name	Identifies a camera in a multi-camera deployment.
Simulation mode	Internal function; leave disabled.
Image-save mode	Saves algorithm image data on the camera.
Image-acquisition mode	`MDS` does not occupy camera access. `SDK` does; the host application cannot open the camera while the algorithm is running.
`x`, `y`, `z`	Camera extrinsic position; `z` is the camera height above the ground.
`roll`, `pitch`, `yaw`	Camera extrinsic orientation angles.
Ideal ground	Keep the default value.
Read configuration from camera	Reads the current camera configuration.
Save and send	Writes changes to the camera. Unsaved parameters do not take effect.
Extrinsic calibration	Calibrates camera height and orientation.

Vertical installation
Horizontal installation

Open Camera configuration. Before calibration, set x, y, z, roll, pitch, and yaw to 0, then select Save and send.
Select Extrinsic calibration. After calibration succeeds, verify the result in the parameter panel.

Extrinsic calibration for a vertical installation

Open Camera configuration and enter the parameter-calibration page.
Align the point-cloud ground with the coordinate system: the red X and green Y axes lie on the ground, while the blue Z axis is perpendicular to and points up from the ground.
Usually adjust only roll and z first; set the other values to 0. Select Save and send after verification.

Extrinsic calibration result for a horizontal installation

Algorithm parameter reference

Select Save and send after changing any algorithm parameter. Keep filter parameters at their defaults unless an on-site test shows that adjustment is required.

Human detection
Storage-location detection
Volume detection

Parameter	Description
Algorithm version	Current algorithm version.
Detection frequency	Refresh frequency for images and results.
Temporal filter	With `3`, three consecutive identical results are required before a result is considered valid.
Kernel size	Kernel size for 2D morphological operations. Larger values shrink targets and expand the background.
Distance threshold	Determines whether points are similar or belong to the same class.
Minimum point count	Minimum point-cloud count required for semantic clustering.
Minimum non-semantic point count	Minimum point-cloud count required for non-semantic clustering.
Neighborhood size	Defines point-neighborhood range and affects clustering granularity.
Rotation detection	Match the camera installation direction; use the logo-down orientation as the `0°` reference.

Region parameter	Description
Add / delete region	Create or delete a human-detection region.
Current box	Select the active detection region.
Minimum / maximum `X`, `Y`, `Z`	Define the region’s 3D boundaries.
ID	Region name used to distinguish multiple regions.
Drag to adjust region	Drag with the left mouse button along `x` and `y`; drag with the right button along `z`.

Global storage-location detection configuration

Parameter	Description
Algorithm version	Current algorithm version.
AI filter	Semantic segmentation; not currently available.
Detection frequency	Result refresh frequency.
Alignment angle	Maximum allowed cargo angle offset when alignment checking is enabled.
Alignment check	Checks angle offset and boundary status. Detected objects are shown with yellow volume boxes.
Temporal filter	Outputs a valid state only after several consecutive consistent results.
Minimum point count	Objects with fewer points than this value are ignored.

Region mode	Configuration
2D RGB	Use minimum and maximum pixel `X` and `Y` to define the storage location. Semantic mode and a semantic model are required.
3D point cloud	Use minimum and maximum `X`, `Y`, and `Z` to define the location. Drag with the left mouse button along `x` and `y`, or with the right button along `z`.

Advanced feature	Description
Copy current box	Copy the active storage-location region.
Direction, spacing, count	Define evenly spaced copies along the `X`, `Y`, or `Z` axis.
Run	Generate multiple regions from the configuration.
QR Code calibration	Enter the location width, length, `Z_MIN`, and `Z_MAX`, then calibrate with a QR code.

3D storage-location region configuration

Parameter	Description
Algorithm version	Current algorithm version.
Trigger mode	When enabled, acquire one image and result for each trigger.
Upward detection	Detect using the reverse camera coordinate system.
Detection frequency	Refresh frequency for images and results.
Detection-plane distance	Reference position of the object’s lower surface, in mm.
Neighborhood count / distance threshold	Filter parameters; keep the defaults in most deployments.
Normal-vector filter threshold	Filters tilted point-cloud planes. The original recommendation is `60`.
Normal-vector distance threshold	Filter parameter. The original recommended range is `20–30`.
Minimum point count	Objects with fewer points than this value are ignored.
Minimum `X`, `Y`, `Z`	Minimum length, width, and height thresholds, in mm.

Region parameter	Description
Minimum / maximum `X`, `Y`, `Z`	Defines the 3D boundaries of the volume-detection region.

Deploy an algorithm

Switch to point-cloud mode and scroll over the image area to display point-cloud data.

Human detection

Calibrate a horizontal installation

Align the X and Y axes with the ground point cloud. The Z axis must be perpendicular to and point up from the ground.

Set global parameters

Open Human detection. Usually only Rotation detection needs to be set. Clockwise camera rotation supports 0°, 90°, 180°, and 270°.

Create the detection region

Select Add detection region, set minimum and maximum X, Y, and Z, and move the region over the target. The original example uses a 1000 × 1000 × 2000 mm region.

Save and verify

Select Save and send, run a person-through-area test, and verify the result in the status bar or historical playback.

Matching rule	No person	Person	Obstacle	Person and obstacle
Depth only	`0`	`2`	`2`	`2`
Semantic plus depth	`0`	`1`	`0`	`1`

Depth-only mode reports every object in the region. Semantic-plus-depth mode reports only clusters classified as people.

Storage-location detection

Calibrate the camera

Follow the extrinsic-calibration procedure. Align X and Y with the ground and keep Z perpendicular and pointing up.

Configure global parameters

Set Alignment check, Alignment angle, and Minimum point count for the site. Enable alignment checking when cargo must be placed at a defined angle.

Create storage locations

In point-cloud mode, select Add storage location, switch to the top view, and press X to start or stop drawing. Drag the rectangle over the pallet and adjust the X, Y, and Z boundaries. Press X again to exit drawing mode.

Save and verify

Select Save and send. Configure locations in 2D RGB or 3D point-cloud mode and check Result in the status bar.

Drawing a storage-location region in point-cloud mode

Storage-location state	`Result`
Empty	`0`
Occupied, or occupied and aligned	`1`
Occupied with an angle beyond the alignment threshold	`2`
Occupied and outside the location boundary	`3`

For a regular layout, use Copy current box to create locations at a fixed direction and spacing. If QR-code verification is required, use QR Code calibration to configure location dimensions and the depth range.

Volume detection

Calibrate a vertical installation

Volume detection works from top to bottom. After calibration, the Z axis points down and the X/Y plane is parallel to the ground.

Set the detection plane

For goods placed directly on the floor, set the detection plane to the floor height. To measure only goods on a pallet, align the plane with the pallet point cloud.

Set minimum dimensions and the region

Set minimum X, Y, and Z values to filter small targets. Then set the region boundaries according to the detection plane and cargo dimensions.

Save and verify

Select Save and send, place one or more objects in the region, and verify the returned length, width, height, and count.

Floor detection plane for volume detection

To use single-trigger acquisition, enable Trigger mode in Volume detection and select Save and send. Then select Single trigger in Basic tools, or send TM_TRIGGER over TCP or UDP.

Communication interfaces

The IP address in the examples is illustrative. Replace it with the camera address at the site, then verify the network, algorithm state, and result format before integrating with an upper-layer system.

Transport methods

Protocol	Port	Request	Use
TCP	`14951`	`RESULT_UPDATE`	Get the current algorithm result.
UDP	`14950`	`RESULT_UPDATE`	Get the current algorithm result.
HTTP	`14952`	`RESULT_UPDATE`	Get the algorithm result as JSON.
TCP / UDP	Result port	`TM_TRIGGER`	Run one triggered volume detection and return the result.
Modbus TCP	`16502`	Read holding registers	Read storage-location status.

HTTP result request

The upper-layer system sends a result request to the vision endpoint, which returns a JSON string. Use the project-delivered request headers in your HTTP client and the following request body.

string

default:"RESULT_UPDATE"

required

Triggers the camera to return the current algorithm result.

Common response fields

string

required

IP address of the camera that produced the data.

string

required

Camera name; change it in camera configuration.

integer

required

Return code; 0 indicates success.

Human detection (SG)
Storage-location detection (SS)
Volume detection (TM)

object

required

Status object keyed by detection-region ID: 0 means no target, 1 means a person was detected, and 2 means an object was detected only by the depth rule.

string[]

Cluster details. Each item is a JSON string that must be deserialized again.

string

Human-detection data timestamp.

string

Human-detection result timestamp.

Cluster details include rbx2d, rby2d, tlx2d, and tly2d (2D box coordinates), semantic_index, traceID, and x_min, x_max, y_min, y_max, z_min, and z_max (3D box boundaries).

object

required

Status object keyed by location ID: 0 empty, 1 occupied, 2 occupied with an angle offset, and 3 occupied and out of bounds.

number[]

Details for location X: center x, y, and z, length, width, height, and angle offset from the bounding box. A location created in 2D returns all -1 values.

string

Storage-location data timestamp.

string

Storage-location result timestamp.

object[]

required

Array of complete volume boxes in the detection region.

number

Volume-box angle offset.

number

Volume-box center x coordinate.

number

Volume-box center y coordinate.

number

Volume-box center z coordinate.

number

Object length.

number

Object width.

number

Object height.

string

Semantic type.

boolean

Whether the result is valid.

string

Volume-detection data timestamp.

string

Volume-detection result timestamp.

Human-detection response example

{
  "IP": "192.168.100.83",
  "SG": {
    "1": 1,
    "2": 0,
    "__object_details": []
  },
  "SG_DATA_TIME": "2023_03_02_21_52_11.408",
  "SG_RES_TIME": "2023_03_02_21_52_11.689",
  "nick_name": "Unknown",
  "ret_code": 0
}

Modbus TCP (storage-location detection)

Results are ordered by storage-location ID; at most the first 16 locations are returned.

Configuration	Value
Port	`16502`
Listen address	`0.0.0.0` (all network interfaces)
Register count	`16`
Start address	`0x0100`
Byte order	Big-endian

Register	Content	Data format
`0x0100 + ID`	Storage-location and marker detection results	2-byte `uint16_t`
High 8 bits	Storage-location status code	`0–2`
Low 8 bits	Marker status code	`0–2`; this feature is not currently available, so it has the same meaning as the location status code.

Location status	Meaning	Condition
`0`	Empty	No goods or pallet.
`1`	Occupied normally	Goods are placed as required.
`2`	Occupied abnormally	Cargo size or position is outside the requirement.

Marker status	Meaning
`0`	Marker not detected.
`1`	Marker detected.
`2`	Marker detection error.

Read the fourth storage location

The following example uses transaction ID 0x0001 to read one holding register at address 0x0104. The response contains status 0x0002.

Request: 00 01 00 00 00 06 FF 03 01 04 00 01
Response: 00 01 00 00 00 05 FF 03 02 00 02

Field	Request value	Description
Transaction identifier	`00 01`	Client sequence number; the response must match it.
Protocol identifier	`00 00`	Fixed Modbus/TCP value.
Length	`00 06`	Number of bytes in the following request fields.
Unit identifier	`FF`	Broadcast mode or device address.
Function code	`03`	Read holding registers.
Start address	`01 04`	`0x0100 + 4`.
Register count	`00 01`	Read one register.

Troubleshooting

Symptom	Priority checks
Camera not found	Confirm that the host and camera share a subnet, the network adapter is gigabit, and power and switch links are healthy.
Parameters do not take effect	Confirm that Save and send was selected after the change.
No algorithm result	Confirm that the correct algorithm firmware was flashed, the device is green, and the algorithm is enabled.
Empty or stale point cloud	Scroll over the point-cloud area to display data; enable automatic refresh if needed.
Abnormal location or volume results	Recheck extrinsic-axis directions, detection plane, and region boundaries, then retest with site objects.

Getting started

Software tools

Algorithms & deployment

Deploy detection algorithms and integrate results

Supported scope

System components

Algorithms and cameras

Host application environment

Hardware and network deployment

Install the camera

Connect the devices

Configure the host network

Install and upgrade software

Install the host applications

Upgrade algorithm firmware

Algorithm host application

Startup and status

Basic tools

Status bar and playback

Extrinsic calibration

Camera configuration

Algorithm parameter reference

Deploy an algorithm

Human detection

Storage-location detection

Volume detection

Communication interfaces

Transport methods

HTTP result request

Common response fields

Human-detection response example

Modbus TCP (storage-location detection)

Read the fourth storage location

Troubleshooting

​Supported scope

​System components

​Algorithms and cameras

​Host application environment

​Hardware and network deployment

​Install the camera

​Connect the devices

​Configure the host network

​Install and upgrade software

​Install the host applications

​Upgrade algorithm firmware

​Algorithm host application

​Startup and status

​Basic tools

​Status bar and playback

​Extrinsic calibration

​Camera configuration

​Algorithm parameter reference

​Deploy an algorithm

​Human detection

​Storage-location detection

​Volume detection

​Communication interfaces

​Transport methods

​HTTP result request

​Common response fields

​Human-detection response example

​Modbus TCP (storage-location detection)

​Read the fourth storage location

​Troubleshooting

Supported scope

System components

Algorithms and cameras

Host application environment

Hardware and network deployment

Install the camera

Connect the devices

Configure the host network

Install and upgrade software

Install the host applications

Upgrade algorithm firmware

Algorithm host application

Startup and status

Basic tools

Status bar and playback

Extrinsic calibration

Camera configuration

Algorithm parameter reference

Deploy an algorithm

Human detection

Storage-location detection

Volume detection

Communication interfaces

Transport methods

HTTP result request

Common response fields

Human-detection response example

Modbus TCP (storage-location detection)

Read the fourth storage location

Troubleshooting