When starting with cameras in embedded systems, one might think of the camera primarily as a device from which we simply receive an image. We plug it in, display the image, and add a few settings, like exposure or white balance. In practice, someone working with a camera will quickly realize it’s a bit more complicated, especially after taking a quick look at camera settings.

The Camera as Part of a Larger System

In WizzDev, we worked on a digital microscopy system running Embedded Linux. The camera was just one of several elements of the entire device. We simultaneously had to display the live image, handle the GUI settings, and ensure what the application showed matched what was accepted by the camera. Simply put, whether the signal sent to the camera matches how the camera was supposed to react.

At first, it all may sound a bit complicated. It’s much easier to understand the whole system when we divide communication with the camera into two separate parts:

  • Video path – responsible for transmitting the image,

  • Control path – responsible for changing camera settings and receiving responses.

Both paths work at the same time in the camera, but they have completely different tasks. Starting the image display itself is therefore only the first element of a more complex system.

Image and camera control in embedded systems

The camera image is transmitted almost continuously. It can reach the device via MIPI CSI-2, Ethernet, or the commonly known USB cable.

In an Embedded Linux system, the application usually does not communicate directly with the hardware. There are a few layers between the application interface and the camera. According to the NVIDIA Jetson Linux: Camera Software Development Solutions, these could include V4L2, GStreamer, or libargus, depending on the camera configuration.

However, camera control looks different. Here we send rather small commands, for example:

  • change exposure,
  • set gain,
  • change white balance mode,
  • execute One Push White Balance.

In one of our projects, the image and commands were transmitted over separate connections. The image had its own path, and camera control took place via UART. Another separate connection allowed the application to independently receive the image and change camera settings.

What are MIPI, V4L2, and UART?

When starting your journey with camera control in embedded systems, it’s easy to mix up these terms. So let’s start from the beginning.

Firstly, MIPI CSI-2 is a high-speed communication protocol for the physical transmission of image sensors to the CPU. It is commonly used for cars, drones, and many other camera systems.

Secondly, there is Video4Linux2. V4L2 is an interface available in Linux that allows applications to use video devices and some of their settings. While MIPI CSI-2 is responsible for sending electrical signals from the camera sensor to the processor, V4L2 transforms those signals into frames and passes them to the app.

And last but not least, UART. UART is a simple communication protocol used by devices for basic communication. It can be compared with MIPI CSI-2 as another device component working as hardware. But as MIPI is a five-lane motorway, UART is just a regular countryside road.

Different tasks, one device

Through V4L2, you can, for example, correct exposure, gain, or white balance if a given camera driver supports it. V4L2 itself also distinguishes between automatic and manual modes. According to The Linux Kernel: Camera Control Reference documentation, automatic and manual exposure are defined separately. With automatic exposure active, the driver may even ignore a manual exposure time change.

Similarly, V4L2 distinguishes continuous automatic white balance from a one-time operation, after which the camera retains the calculated setting.

Not every camera function may be available through V4L2. In a more custom device, the camera may have its own communication protocol. The application can then receive the image via a standard video interface and send more specific commands via UART, for example. We therefore communicate with one camera in two different ways.

GUI state and camera state

A change in the GUI and the completion of an operation by the camera do not have to happen at the same time. If the user changes a setting, the application may immediately show the new value. This does not mean the camera has actually set it yet. This is one of the most crucial things to learn about camera control in embedded systems.

One Push White Balance example

The user clicks a button, the application sends a command, but the camera needs some time to perform the calibration. The whole operation can look something like this:

The user clicks a button → the application sends a command → the camera starts calibration → the camera informs about completion → the application reads the result → the GUI is updated

If the application only recognised the GUI, it could consider the operation complete just after the button click. In an embedded system, it’s better to wait for confirmation from the hardware. From the user’s perspective, this is one button in the GUI, but on the application side, there may be several steps of communication with the hardware behind it.

Camera settings

Camera settings can also depend on each other. Manually changing the exposure might not make sense if automatic exposure is still active. The same goes for white balance or gain.

The GUI shouldn’t just store a number. The current camera mode is also needed; that’s why you have to know:

  • whether exposure works automatically,

  • whether the user can now set it manually,

  • which white balance mode is active,

  • what should happen when the camera is reconnected.

While working on the GUI, I also noticed that it’s worth separating the value the user wants to set from the value confirmed by the camera. If the GUI immediately changes the value on the screen and the camera doesn’t accept the command, the user will see something different from what is actually happening in the hardware.

If the camera calculates some value, it’s good to later read it directly from the hardware. After performing a One Push White Balance, the camera can calculate the appropriate gain values itself.

The application can wait for the operation to finish and save these actual values, instead of only saving the information that the user pressed a button. Then the application’s state better corresponds to what is actually happening in the device.

Timeout, ACK, and Command Order

Most problems you might encounter don’t directly stem from transmitting the image. More often, you may run into:

  • timeouts,

  • wrong command order,

  • camera disconnection,

  • reconnection,

  • The camera being busy executing a previous operation.

A typical example is sending a command and getting no response within the expected time. Normally, the camera might respond very quickly, so it’s easy to assume it will always be like that.

The problem arises when the camera is busy or has been disconnected. The application must then know what to do with such a command and what to show the user.

Problems can also arise when the user rapidly changes several settings. The application must know whether the camera can handle several commands at once or if they need to be sent sequentially. Some operations return a response only after some time, so you also need to know which previous command a given response relates to.

In such situations, logs are very helpful. The message “White balance not working” alone says much less than a set of entries like this: command sent -> WB ACK received -> no information about operation completion

In this example, an ACK has already been received, but information about operation completion is still missing. An ACK itself does not mean completion. The logs immediately show whether a problem occurred at any stage.

Reconnect

After the camera is reconnected, the application might still remember old settings, while the camera might have restarted and returned to default values. The GUI could then, for example, show Exposure: 100, while the camera, after a restart, is actually using Exposure: 50.

Both sides are working correctly, but they have different states. After a reconnect, it’s therefore good to read the settings from the camera again or send the configuration from the application to it in a controlled manner.

Communication logic outside the GUI

While working on such a system, you will come to the conclusion that the GUI should not directly handle the entire camera protocol. The GUI should rather say: set exposure or start white balance calibration

A separate part of the application should:

  • prepare the command,

  • send it to the camera,

  • receive the response,

  • check if the operation has finished,

  • handle a timeout or error.

In such a setup, the control path oversees commands, responses, timeouts, and whether the values visible in the GUI actually correspond to the camera settings. The GUI code remains simpler, and communication with the camera is easier to test. Overseeing this communication is one of the most important parts of working with cameras in embedded systems.

Testing whole sequences

Testing a single command might work correctly, but it doesn’t mean the whole system will work well. Problems often appear only when several things happen at once, for example:

  • rapid changes to several settings,

  • camera disconnection during an operation,

  • reconnect,

  • switching between automatic and manual modes,

  • no response to one of the commands.

Besides individual functions, it’s therefore worth testing whole sequences. In embedded, this is especially important because the application is constantly communicating with real hardware.

Our article Best Practices for Embedded Software Integration Testing describes a similar approach. In integration testing, it’s important to check the code and communication between software and real hardware, for example via UART, I²C, or SPI.

Such tests can be partially automated. The article Automated integration testing in embedded systems shows an example of tests written in Python that send commands via UART and later analyze device logs. They are particularly useful for whole sequences and problems that are difficult to reproduce regularly by hand.

Conclusion

Separating video and camera control in embedded systems is essential for building reliable vision applications. Decoupling high-speed frame transport from device configuration keeps the video pipeline fluid. Managing camera settings over a dedicated communication protocol ensures that UI updates reflect actual hardware states rather than unconfirmed commands. 

Unexpected disconnections and timing edge cases disappear when control logic handles state synchronization directly. Mastering camera control in embedded systems ultimately turns unpredictable sensors into dependable components.

Key takeaways

  • The video path transmits the image, and the control path handles settings and the current state of the camera.

  • Changing a value in the GUI does not yet mean the camera has operated. It’s worth waiting until the hardware confirms its completion.

  • After a reconnect, the application and the camera might have different settings, so their state should be synchronized again.

  • In tests, it’s worth checking whole sequences, for example, several rapid settings changes, a timeout, or a camera disconnection during an operation.