During my thesis I developed a multi-sensor data recovery system incorporating video, radar, gps and inertial measurement sensors to capture real data in an automotive context.

This system was developed in order to validate the results obtained through simulations for the estimation of some parameters, that we think, are of invaluable importance when developing security applications for the vehicle industry. Among these parameters we might find the acceleration and speed vector from any vehicle ahead of us.

In order to find these estimates we developed an algorithm for robust derivation through state reconstruction of noisy numerical signals. A paper on the subject is to appear on 2009.

The system is composed of:

• One High Definition webcam for video recording.
• Two Garmin 18 LVC GPS units, the main reason for this will be explained latter.
• One MotionNode inertial mesurement unit (IMU). If you have the money, do not buy this IMU. We experienced a lot of problems with the euler angles due to some kind of divergence on the filtering process. This product is unreliable and support didn’t even care to answer back!. I recommend the Xsens MTi.
• One TRW AC10 radar unit.
• One WiFi Access Point + external antenna. We used this to receive the GPS coordinates of another similar unit (installed in another vehicle), in order to test the GPS based pointing system presented in link.
• One Pan And Tilt for video tracking. The webcam is installed on top of it and it’s supposed to follow another vehicle that transmits its GPS coordinates to us.
• One Labjack UE3 in order to command the Pan And Tilt.
• One powered USB hub.
• One Traco power converter 12 VDC-5 VDC. These are excellent converters.
• One Vector CancardXL. This is a PCMCIA card with two independent CAN channels (nodes). We used both to communicate with the radar and the vehicle CAN buses. The SDK and the documentation provided are more than satisfactory. Very good product.
• Two average laptop computers.
• A lot, a lot of cables .

This is a simplified diagram of the system:

Two independent CAN buses were used since the radar requires some information from the vehicle (speed and steering angle) that is not necessarily coded as the radar expects. One channel of our CAN card was directly connected to vehicle CAN bus through the ODBII maintenance jack. A lot of reverse engineering was necessary since the identifier of the CAN frame that contain the speed and the steering angle of the vehicle, as well as the precision of these sensors, is not something you will find in the vehicle user guide.

This information, once gathered, is recoded and sent to the radar through the second channel of the CAN card connected to the radar CAN bus. The radar is continuously sending information through this bus, which is registered using a TRW proprietary software (Wacci).

Two laptops were used since there were a lot of sensors to be handled at the same time. The CAN architecture previously described was already enough to keep an average computer busy. We decided to separate the radar stuff from the rest of the sensors. However, since now we use two separate computers, we needed to sync those databases somehow. For this reason, we used a GPS on each laptop instead of only one, to record our GPS coordinates. We took advantage of the fact that when synced with the satellites, both GPS will share the exact same timestamp at the same exact time. Each minute, the internal clock of each laptop is updated using the timestamp provided by each GPS.

The video recording process was handled by a code developed by myself using the OpenCV library. Our GPS coordinates were recorded as well as the rotation speed and accelerations on each axis. The euler angles were also registered.

The code was written in C++, and contains multiple threads, each one handling a device of its own. The main reason for this choice is that we didn’t have any; all the SDKs were available in C/C++. Since each sensor do not necessarily provides data at the same time, the information is recorded on different text files with its corresponding timestamp. The time correspondence is then made offline by the GUI developed to exploit all this data.

In order to exploit all this information I developed a GUI (some sort of control panel) in python using PyQt. This interface allows to easily reproduce the recorded scenario and to test our algorithms over real data, the main reason for which all this was done.

I provided a demo video of this GUI in action. I even launched the vector speed estimation (one of our developed algorithms) that gives the orientation of all the vehicles ahead. You will clearly see this in the radar widget; the little cars (tracks) will have a line that indicates the orientation of their speed vector as seen by us. After launch, the algorithm takes some milliseconds to converge. You’ll also be able to see the trace (the little dots behind each vehicle) of each track. The results presented here can be improved; the data used for the estimation of the speed vector is not filtered.

I recommend using the full screen option in the video controls. By using this option you will able to see all the numeric values and details of the GUI.