LPR Technology
The baseline specifications of a License Plate Recognition system, sometimes called ANPR (Automatic Number Plate Recognition), is of course that it should be able to read the number from a vehicle’s license plate. However depending on the usage scenario, many different implementations are possible using different types of camera hardware, image processing resources and network connections.
In the following image, the prototypical block diagram of a License Plate Recognition system is shown:
The system in the image is really the block diagram of the Waysight Camera dedicated for LPR, but roughly the same building-blocks are present when for example using split setup where the camera is separate from the computer doing the image processing or when a generic digital surveillance camera (like the Axis cameras with the Waysight LPR ACAP application) is used for making an integrated LPR system.
Typical LPR camera hardware
The components in a typical LPR camera are:
Lens: The camera lens for LPR cameras for roadside use often needs to have a long focal length since the cameras are mounted on road bridges or poles. In parking lots or garages, the lens can have a significantly shorter focal length. The “speed” of the lens (the f-number) is not very crucial for cameras mounted with infrared illumination, but are important for night-time use without artificial illumination to capture as much light as possible from the plates to be read.
Illumination: Some sort of light is needed to illuminate the vehicle and the number plate. Since daylight is scarce at night, and roadside illumination can be non-existant or very weak, most LPR setups that require 24/7 reading are used with infrared illuminators (usually IR LED arrays either integrated with the camera or placed externally), eliminating the need for any existing light at all. In fact, the less existing light that enters the image sensor the better, since sunlight can cast shadows over plates or reflect inside the lens in bad ways. Nevertheless, for many usage scenarios, existing light is enough and allows the use of more simple surveillance cameras for LPR. Using normal light has the added benefit of letting the camera capture images of the drivers, as IR light is usually too weak to show the driver through the windshield which might be important in some applications.
Image sensor: The sensor forming the digital image needs to capture images with no visible lag over the image since high-speed vehicles produce a quickly moving image of the plate on the sensor. Typically, the main problem here is the widespread use of so-called “CMOS Rolling Shutter” sensors in digital surveillance cameras, which read out the image row-by-row in a relatively slow way, distorting the imaging of high-speed objects. Also, rolling shutter sensors cannot be synchronized to the infrared illuminator, reducing the efficiency of the setup and significantly decreasing the illumination possible for a given size or power budget. In the Waysight LPR Camera, a high-speed global-shutter industrial vision sensor is used with exposure times as low as 1/4000 of a second. If the use-case involves slower moving cars like inside a city, at a red-light, at a parking lot or a gate this is not a problem though.
CPU/processor: The captured images need to be processed to locate candidates of license plate regions and these have to be analyzed to give the resulting text string of the license plate. If this processing is done inside the camera by a dedicated CPU, the setup is usually referred to as a fully integrated LPR camera system or as having “processing at the edge”. One benefit of such a system is scalability – since the processing happens on board each camera in a system, the network connecting the cameras doesn’t need to carry HD video streams and the server that logs the resulting data only receives small chunks of text and doesn’t have to receive, unpack and process video. Another benefit is for privacy reasons the processing and decision making (opening a gate for example) has to stay on the camera.
Many applications call for cameras to be installed at locations without physical network connections, using 4G wireless connections, and in such a setup it is not practical or possible to send video streams. Sometimes an in-between solution is used where a co-located PC is used to process a small number of video streams from local LPR cameras.
Network connections: The LPR system has to output and transfer the read plates and/or the images of the plates to another computer or server where further statistics or logging/searching can take place. For physically connected systems, this is usually done either using Ethernet or RS-485 and for wireless setups a 4G GSM modem is either built into the LPR camera or attached nearby.
Image processing algorithms
The LPR image processing algorithm itself is a piece of software which is run on the CPU in the camera, in an external PC, or in a network server receiving the video streams. It consists mainly of these parts:
License plate detection: The image is quickly scanned and image locations are scored for the possibility of finding a license plate. Regions that have a high probability of containing plates are marked for further analysis. Depending on implementation, these scanning algorithms can sometimes be fooled by image areas containing a high number of details with high contrast, such as trees and roadside bushes. A common recommendation is to place the camera pointed slightly downwards so that only the road surface is visible when there are no vehicles passing. This also prevents seeing vehicles behind the vehicle to be read, which can be important in a gate opening or parking lot scenario.
Character segmentation: The plate candidate regions are segmented into individual characters and the probability that they represent an actual license plate is again evaluated. There can be many regions on a vehicle that contain characters, like country code letters, the vehicle brand, company decals, or simply graphical details that superficially resembles characters as they have the right size. Knowledge of the type of character layouts expected helps this algorithm.
Character recognition: The segmented characters are recognized and turned into codes representing numbers and letters and confidence levels for each possible read. A good character recognition algorithm is flexible and does not rely on a particular typeface and therefore doesn’t need extensive pre-training on slightly varied typefaces. In practice, some optimizations are commonly done when information about which countries number plates are most probable for the LPR camera to see is available.
The image resolution of the incoming plate can play a big role here. If the resolution is low, most characters might then be read correctly but some characters in particular gets difficult to discern like D and O or 5 and S which might look very similar in a low-res image.
Final plate read assembly: The recognized plate character strings are evaluated for their probability to represent a real license plate. This is helped by having some knowledge of the syntax or typical number of characters of a license plate in the desired country’s plate system, as some characters might have been read ambiguously. Reads from many images of the same vehicle might be combined in this step to raise the confidence level of a read.
While an LPR algorithm can in theory read any string of any typeface on the vehicle, some prior knowledge is necessary on how the layout and syntax of the expected license plates look like, otherwise the algorithm ends up detecting all kinds of other text strings that might be printed on the vehicles. When the LPR system contains its own high-power infrared illuminator, this problem is reduced since all items on the vehicle which are not retro-reflective (like number plates usually are) will have a very low brightness in the image.
Many LPR systems are divided into many physical boxes, containing the illuminator, camera, processor and power supply, requiring a lot of wiring and installation efforts at the site, especially considering this all has to be weather proof. The practice of using off-the-shelf industrial computers, cameras and illuminators might lead to quicker design times, but also lead to high power consumption, often in the range of 50 W and upward, making mobile installations or solar powered installations impractical.
As a concrete example, in the integrated approach taken in the design of the Waysight LPR Camera, all components are contained in the same box, and the use of highly optimized image processing software on a low-power ARM CPU reduces the power consumption to below 10W, allowing 24 hrs of mobile continous operation from a small pack of batteries. Also, a typical solar panel can generate around 15 W, thus making completely isolated system installations possible.
Reading performance
At the core of a good license plate recognition system is a robust character segmentation and recognition engine as outlined above. Even if the illumination and camera systems are well designed, the resulting image of a number plate can often be far from ideal. The following sections give some examples of the problems typically encountered which an LPR system has to handle. Most of the situations pose the same problems both using infrared and normal light.
Examples of potentially problematic license plate images
The images below are from live installations and thus have been modified to protect the integrity of the imaged drivers. All images were correctly read by the Waysight LPR system.
Dirt and broken surfaces
Many vehicle plates are covered with dirt or are broken or cracked in some way, or screws and bolts holding the plate to the vehicle are badly placed, touching the characters.These effects pose additional problems for the character recognition algorithms. A system might work fine during summer but fail easily during winter when snow and dirt build up on the plates. The Waysight LPR algorithms have been extensively tested and optimized during the cold winters of Sweden.
Background graphics
Number plates can have background graphics that can confuse both the segmentation and recognition of individual characters in the plate.
Varying reflectance
When plates come new from the factory they have a good, flat reflectance to incident infrared light. Over time this property can degrade overall or in an asymmetrical way, and some vehicle owners actually try to alter the infrared reflectance property to avoid being scanned by tollgates. This effect is less noticeable for visible-light systems.
Varying exposure level
Because of varying distance and angle to the camera the illumination of the plate and therefore the returned light varies in intensity, from dark but properly imaged plates to overexposed plates that have bled out causing an apparent thinning of the characters. Part of this effect can be reduced by a high dynamic range camera system (HDR) but some of the variance still has to be handled by the character segmentation algorithm.
Partial obstruction
Often rear plates are chosen for capture because there are no strong headlights that can confuse the cameras (see below). Sometimes this can cause problems though because of the practice of drivers hanging equipment on the backside of the car, like cargo containers or bicycles, potentially obstructing the rear plate.
Narrow or wide characters
The width of the characters and thus the width of line segment features can vary a lot between different national number plate systems. Typically in the US, the plates are more narrow while in Europe the plates are wider. The zoom-setting on the lens and the distance from the camera to the vehicles have to be adjusted so that the captured image of the most narrow plate required to be read still projects to the minimum pixel width needed by the LPR-algorithm in the image.
Tilt
Most cameras are not installed perfectly level, so the system has to be able to read at least partially rotated images.
Headlights
If the LPR camera is mounted to face the front of the vehicles, the headlights can either disturb the dynamic range of the camera or at least cause structure in the image which can be confused with the number plate, for example if a vertical structure adjacent to the plate looks just like the number 1 or letter I. This effect is more noticeable during night, but can be compensated by having more powerful infrared illumination in the camera, making the plate more visible compared to the headlights. Many highway installations chose to capture the back of the vehicles instead.