A Non-contact Method for Sensing Tire Contact Patch Deformation Using a Monocular Vision System and Speckled Image Tracking by Russell Wright Green A thesis submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Master of Science Auburn, Alabama August 6, 2011 Keywords: tire, deformation, optical, digital image, Lucas-Kanade Copyright 2011 by Russell Wright Green Approved by Song-yul Choe, Co-chair, Associate Professor of Mechanical Engineering Robert L. Jackson, Co-chair, Associate Professor of Mechanical Engineering David Bevly, Professor of Mechanical Engineering ii Abstract A full-field, non-contact deformation sensing system was developed with an application for tires. Using an image tracking algorithm, in-plane displacements and their gradients were calculated. Furthermore, the effects of blurring from out-of-plane displacement and illumination variation were included in the algorithm developed, providing more accurate in-plane information. An imaging system was placed inside a tire that was compressed by approximate normal loadings. Images of the compressed inner liner were recorded and processed. Although a lens defect made independent determination of the out-of-plane displacement to blurring relationship impossible, an assumption was made given the normal loading condition that allowed normal strains to be plotted at six increments of vertical deflection. The longitudinal strain distribution shows an interesting behavior hardly noted in most literature although the tensile strain does generally increase with increasing load. Having proved that digital image processing can measure tire deformation accurately, this research should provide a solid foundation to develop the technique presented into more robust and efficient forms. With improvements, this technology could be implemented in real automotives to provide the high fidelity information needed to derive accurate tire parameters for advanced electronic stability control systems. iii Acknowledgments I would like to give much thanks to my loving parents Daniel and Paulette Green who have always provided me guidance and supported my decisions throughout my life. I would also like to thank my good friends David Jones and Taylor Wingo for listening to my ramblings about my research and providing new perspectives and opinions. I dedicate this work to the late Tugaloo Green who was the greatest feline companion I ever knew. iv Table of Contents Abstract ............................................................................................................................... ii Acknowledgments.............................................................................................................. iii List of Tables ..................................................................................................................... vi List of Figures ................................................................................................................... vii Nomenclature .......................................................................................................................x Chapter 1: Introduction ........................................................................................................1 Chapter 2: Background and Literature Review ...................................................................2 2.1 Pneumatic Wheel Anatomy .............................................................................. 2 2.2 Tire Deformation Overview .............................................................................. 3 2.3 Current Sensing Technologies .......................................................................... 5 2.4 Motivation ....................................................................................................... 17 Chapter 3: Theory ..............................................................................................................19 3.1 Proposed Technique ........................................................................................ 19 3.2 Warping Function ........................................................................................... 19 3.3 Image Tracking ............................................................................................... 21 3.3.1 Preliminaries .................................................................................... 21 3.3.2 Subset Size Selection ....................................................................... 22 3.3.3 Forward Compositional Algorithm .................................................. 23 3.3.4 Inverse Compositional Algorithm.................................................... 25 v 3.3.5 Coarse Search and Initialization ...................................................... 26 3.3.6 Levenburg-Marquardt Iteration ....................................................... 26 3.3.7 Sub-pixel Interpolation .................................................................... 27 3.4 Illumination Scale and Offset ......................................................................... 28 3.5 Blur Estimation ............................................................................................... 29 3.6 Complete Algorithm ....................................................................................... 31 Chapter 4: Simulation and Results .....................................................................................35 Chapter 5: Experimental Setup ..........................................................................................45 5.1 Rim and Tire Constraints ................................................................................ 45 5.2 Camera ............................................................................................................ 47 5.3 Lens ................................................................................................................. 49 5.4 Illumination ..................................................................................................... 51 5.5 Mounting ......................................................................................................... 52 5.6 Pressure and Temperature Monitoring............................................................ 55 5.7 Tire Surface Preparation ................................................................................. 55 5.8 Tire Loading.................................................................................................... 56 5.9 In-plane Deformation Correction .................................................................... 57 Chapter 6: Experimental Results and Analysis ..................................................................66 Chapter 7: Discussion and Conclusions .............................................................................88 7.1 Contributions................................................................................................... 89 7.2 Future Work .................................................................................................... 89 References ..........................................................................................................................91 Appendix ............................................................................................................................95 vi List of Tables Table 1: Physical meaning of affine parameters ............................................................... 20 Table 2: First simulation subset values. ............................................................................ 38 Table 3: Second simulation subset values. ....................................................................... 40 Table 4: Third simulation subset values. .......................................................................... 42 Table 5: Fourth simulation subset values.......................................................................... 43 Table 6: IDS Ueye 1466 LE camera specifications. ......................................................... 48 Table 7: Loads corresponding to applied out-of-plane displacements. ............................ 67 vii List of Figures Figure 1: Tire components [4]. ........................................................................................... 3 Figure 2: Wheel axes and planes. ....................................................................................... 5 Figure 3: Tire deformation sensor employing Hall effect [6]. ............................................ 7 Figure 4: Piezoelectric film sensor for measuring lateral sidewall deflections [8]. ............ 8 Figure 5: Ultrasonic tire sensor [9]. .................................................................................... 9 Figure 6: Tire deflection sensor coupled to SAW elements [10]. ..................................... 10 Figure 7: Diagram of PSD sensor detecting IR diode glued to inner liner [12]. .............. 11 Figure 8: Three-axis accelerometer [14]. .......................................................................... 12 Figure 9: Flexible epoxy strain sensor developed for tire use [15]. .................................. 13 Figure 10: Tire strain sensor using natural rubber as the substrate [16]. .......................... 14 Figure 11: Using steel wires of a tire to detect deformation through capacitance changes [17]. ................................................................................................................................... 15 Figure 12: Camera and laser diodes used to capture in-plane and out-of-plane displacements [20]. ........................................................................................................... 16 Figure 13: Diagram of camera imaging the tire surface with rubber blocks attached [21]. ........................................................................................................................................... 17 Figure 14: Affine transformed subset. .............................................................................. 21 Figure 15: Example speckle images. Panel A is the template image and Panel B is the deformed image. ............................................................................................................... 22 Figure 16: Sub-pixel demonstration. ................................................................................. 28 Figure 17: Two dimensional Gaussian example at ? = 2. ................................................. 30 viii Figure 18: Complete algorithm flowchart. The coarse search and inverse update algorithm are presented. .................................................................................................... 33 Figure 19: Complete algorithm flowchart continued. The forward update algorithm is presented. .......................................................................................................................... 34 Figure 20: Simulation result. p5 = 13.5px, p6 = 13.5px. .................................................. 37 Figure 21: Simulation results. ?1 = 0.08 and ?4 = 0.08. ............................................ 39 Figure 22: Third simulation plot. ?2 = 0.2 and ?3 = ?0.2. ........................................... 41 Figure 23: Fourth simulation plot. ?2 = 0.2 , ?3 = 0.2, and ? = 2. ............................... 43 Figure 24: Three piece rim. ............................................................................................... 46 Figure 25: Rim and tire dimensional constraints. ............................................................. 47 Figure 26: IDS Ueye 1466LE camera. .............................................................................. 49 Figure 27: Lensation S-mount lens. The lens chosen has a 3.5mm focal length and made to have a low distortion. .................................................................................................... 50 Figure 28: Four-pin white light LED. ............................................................................... 51 Figure 29: CAD assembly drawing. .................................................................................. 53 Figure 30: Final assembly. Five LEDs were soldered to prototyping board and fastened as shown. The lens and camera are precisely positioned in the center of the mounting plate. ........................................................................................................................................... 54 Figure 31: Pave Technologies hermetically sealed plug. .................................................. 55 Figure 32: Speckled inner liner. ........................................................................................ 56 Figure 33: Tire in MTS Q-Test machine. Note the aluminum head used to ensure load was evenly applied over the entire contact patch area. ..................................................... 57 Figure 34: Blur to displacement calibration procedure. Note the 2.5mm spaced lines between the camera and speckled surface. ....................................................................... 59 Figure 35: Blur parameter surface plot. This plot demonstrates the astigmatism inherent to the lens used. ................................................................................................................. 60 Figure 36: Scale to blur relationship for center subset. .................................................... 61 ix Figure 37: Proportionality of out-of-plane displacement to initial depth ratio to scale factor. ................................................................................................................................ 62 Figure 38: Blur to scale correction flowchart. This procedure would be used if the lens had no astigmatism. .......................................................................................................... 64 Figure 39: Raw pixel displacement at 2mm deflection. ................................................... 68 Figure 40: Longitudinal strain at 2mm deflection. ........................................................... 69 Figure 41: Lateral strain at 2mm deflection. ..................................................................... 70 Figure 42: Raw displacement at 4mm deflection. ............................................................ 71 Figure 43: Longitudinal strain at 4mm deflection. ........................................................... 72 Figure 44: Lateral strain at 4mm deflection. ..................................................................... 73 Figure 45: Raw displacement at 6mm deflection. ............................................................ 74 Figure 46: Longitudinal strain at 6mm deflection. ........................................................... 75 Figure 47: Lateral strain at 6mm deflection. ..................................................................... 76 Figure 48: Raw displacement at 8mm deflection. ............................................................ 77 Figure 49: Longitudinal strain at 8mm deflection. ........................................................... 78 Figure 50: Lateral strain at 8mm deflection. ..................................................................... 79 Figure 51: Raw displacement at 10mm deflection. .......................................................... 80 Figure 52: Longitudinal strain at 10mm deflection. ......................................................... 81 Figure 53: Lateral strain at 10mm deflection. ................................................................... 82 Figure 54: Raw displacement at 12mm deflection. .......................................................... 83 Figure 55: Longitudinal strain at 12mm deflection. ......................................................... 84 Figure 56: Lateral strain at 12mm deflection. ................................................................... 85 Figure 57: Longitudinal strain profile at y = 0mm. .......................................................... 86 Figure 58: Lateral strain profile at x = 0mm. .................................................................... 87 x Nomenclature ? Gaussian standard deviation p Deformation parameter vector ?p Deformation parameter update vector x Coordinate vector, [x y]T ? Illumination offset ?? Change in illumination offset ? Illumination scale ?? Change in illumination scale u X-axis displacement (px) v Y-axis displacement (px) ?? ?? Gradient of u with respect to x ?? ?? Gradient of u with respect to y ?? ?? Gradient of v with respect to x ?? ?? Gradient of v with respect to y T Template image I Deformed image G Two dimensional Gaussian function H Hessian matrix xi W Warping function ?xx Normal strain in x direction (mm/mm) ?yy Normal strain in y direction (mm/mm) res Camera pixel resolution (px) FOV Field of view (mm) ? Gradient operator ?2 Laplacian operator d0 Desired displacement accuracy (mm) dI Image displacement accuracy (px) sfd Sensor format dimension (mm) w Working distance (mm) s Number of speckles Ak Vector of random speckle amplitudes, or intensities xk Vector of random positions in x direction (px) yk Vector of random positions in y direction (px) R Radius of specles (px) Z Initial distance from lens to imaged surface (mm) ?Z Change in distance from lens to imaged surface (mm) ? ?2 ?? Change in ? r Radius of blur (px) ?? ?p Jacobian matrix f Focal length (mm) 1 Chapter 1: Introduction The modern automobile hosts numerous sensors collecting information concerning the vehicle?s surroundings, state, and even its occupants, yet one of the most important aspects is largely overlooked: the tire contact patch. This small patch of tire contacting the road provides the capability for the essential purpose of propulsion and control. Thus, it is not surprising that tire companies are devoting much effort to the development of tire sensing methods and technologies. The US Air Force also sees the importance of sensing the tire contact patch as evidenced by the 2009 SBIR proposal AF093-210 [1] titled ?Aircraft Tire Contact Patch Force and Shear Sensor? in which a sensing technology was sought to measure forces in three dimensions. A typical tire has a complex structure and shape that poses a harsh environment to supplementary electronics. Temperatures range from 20 to 60?C, road conditions or driving styles may impose large deformations, and ambient moisture and shop fluids such as oils and tire soap may enter the tire cavity. Another concern is data transmission given the tire is a rotating component composed of rubber, steel cable, and other materials that attenuate wireless signals. Tires have always been fundamental to vehicle dynamics but their inception as an active sensor was a development largely spawned by the passage of the TREAD act [2]. This mandated the installation of tire pressure monitoring systems (TPMS) in the wheels of all vehicles produced after 2005. As electronic stability control systems become more advanced, having accurate data from the tire contact patch proves invaluable. 2 Chapter 2: Background and Literature Review 2.1 Pneumatic Wheel Anatomy The wheel system is composed of a rim and an inflated tire. Three key features of a standard radial passenger car tire are the tread, sidewall, and bead. Figure 1 displays the important components described in the following discussion. The tread provides the actual interface between the road and tire and is usually characterized by a particular pattern of tread blocks and grooves that aid in road adhesion and water removal. Beneath the tread are several layers including steel belting, nylon plies, body plies, and rubber to hold everything together. The layer of specially formulated rubber coating the tire?s inner surface is called the inner liner, which is meant to reduce air permeation and leakage. The body plies generally consist of nylon or rayon chords that form the shape of the tire. Sidewalls are composed of these plies and rubber. The bead has a central, wound steel cable surrounded in rubber with the body plies wrapping around it. The bead anchors and seals the tire to the rim mounting flange [3]. 3 Figure 1: Tire components [4]. For the purpose of this work, the cylindrical portion of metal that spans the rim?s width will be referred to as the hoop. The hoop is connected to the hub through radiating spokes while the hub allows the wheel to be fastened to the vehicle?s wheel spindle. As previously mentioned, the tire bead fills the bead seating area delineated by the inboard and outboard flanges and inboard and outboard safety humps corresponding to the rim?s outer and inner faces respectively. 2.2 Tire Deformation Overview The tire contact patch experiences radial, longitudinal, and lateral deformation corresponding to each direction in an orthogonal coordinate system. The directions and 4 planes associated with the tire contact patch are defined and illustrated in Figure 2. Radial deformation, or deflection, is a consequence of the vertical, or normal, wheel load that flattens the once curved contact patch to the relatively flat roadway. The radial deformation may vary over the contact patch due to camber. Camber is the tilt of the wheel plane with respect to the vertical axis, and it is formally measured by the inclination angle. Longitudinal deformation occurs in the wheel heading direction and results from the friction between the contact patch and road. Lateral deformation occurs in the direction perpendicular to the wheel heading direction but within the contact plane. It occurs most notably when the tire is steered (when the vehicle is turning). The slip angle measures the angle between the wheel heading direction and the direction of wheel travel [3]. Longitudinal and lateral deformation will be referred to as in-plane deformations while radial deformation will be denoted out-of-plane deformation in this research. 5 Figure 2: Wheel axes and planes. Tire-road friction and slip angle are considered two of the most important parameters to electronic stability control systems [5]. The tire sensing technologies researched thus far attempted to measure some aspect of the tire deformation in order to provide more accurate estimates of those two quantities or provide new information sufficient to derive other parameters for stability control, although those parameters have yet to be defined. 2.3 Current Sensing Technologies Various tire sensing ideas have been explored and tested that can be generally 6 categorized into direct and non-direct approaches. Non-direct approaches include those that attempt to estimate parameters such as vertical load, slip, and camber using existing technologies such as wheel speed sensors or GPS data. These will be excluded from the discussion here in the interest of relevancy to the competing technology being proposed. Direct approaches attempt to measure the tire deformation or forces in the contact patch through displacement, strain, or acceleration emanating at or very near the tread in contact. Furthermore, direct approaches are categorized as contacting (those that interfere with the sensed surface) and non-contacting. Brandt et al. [6] developed and implemented a tire tread deformation sensor using the Hall effect. They implanted four integrated Hall crosses fabricated from AlGaAs/GaAs into a tread block that sensed a permanent magnet just 1mm below the sensor in order to monitor deformation. The authors reported measuring deformations in all three directions [6]. Brandt was also involved in the work of Yilmazoglu et al. that increased the sensitivity and lowered power consumption of the previous sensor by using InAs/GaSb [7]. It was clear that the major disadvantage of their sensors was the necessity to implant it directly in a tread block. Figure 3 shows a concept drawing of this sensor. 7 Figure 3: Tire deformation sensor employing Hall effect [6]. Erdogan et al. [8] developed a lateral sidewall deflection sensor composed of a piezoelectric film affixed to a cantilever beam rooted at the tire bead. The cantilever was attached to the inner-liner via an elastic cylinder that helped to decouple the lateral deflections from radial and longitudinal ones. A lateral deflection would bend the cantilever and induce a measurable voltage. The motivation for this sensor was the fact that slip angle can be related to lateral deflections [8]. Figure 4 shows a schematic of this concept. 8 Figure 4: Piezoelectric film sensor for measuring lateral sidewall deflections [8]. Another unique approach explored measuring radial deflections using an ultrasonic sensor to measure radial deflection. M?gori et al. [9] mounted the sensor to the rim hoop facing the inner-liner so that reflected waves could be recorded thereby providing a measurement distance and hence deflection when the distance changed. However, sound waves could interfere with one another in the tire cavity such that the displacement resolution was low [9]. Figure 5 depicts this sensor concept. 9 Figure 5: Ultrasonic tire sensor [9]. Pohl et al. [10] inserted a metal pin into a tread element that was linked to surface acoustic wave (SAW) elements via a small ball. As the tread element passed through the contact patch, the pin deflected, moving the ball that would bend the SAW?s metal substrate. The SAW elements would then convert the bending into an RF signal that was in turn converted into usable deformation information [10]. Figure 6 below demonstrates this sensor. 10 Figure 6: Tire deflection sensor coupled to SAW elements [10]. A consortium comprising the European Community and several tire and tire related companies funded the project dubbed APOLLO and titled ?Intelligent Tyre for Accident-free Traffic.? This project developed and explored three tire sensing ideas that respectively utilized piezo-resistive strain sensors, an optical displacement sensor, and a tri-axial accelerometer. For the strain sensor concept, two gages were attached to the center of the inner liner, one longitudinally and the other laterally. It was noted that the sensors could only detect deformation in the two directions mentioned and the likelihood of de-bonding posed a serious disadvantage [11]. A. J. Tuononen?s [12] optical displacement sensor encompassed an infrared (IR) diode glued to the inner liner and aimed at a position sensitive detector (PSD) attached to the rim, as shown in Figure 7. Using relatively simple equations, Tuononen was able to determine vertical, longitudinal, and lateral displacements from the IR diode point [12]. 11 Figure 7: Diagram of PSD sensor detecting IR diode glued to inner liner [12]. Pirelli Tire System?s tire sensing brainchild was the use of the tri-axial MEMS accelerometer affixed to the inner liner, as shown in Figure 8. The authors noted that no good correlations could be drawn between the acceleration signals and actual tire mechanical quantities. Therefore, synthetic parameters were derived from the signals that could be used to measure such realistic parameters as contact length and road roughness [13]. The drawbacks of using accelerometers include their dependence on wheel velocity and specific tire construction with the equations used to extract the synthetic parameters being empirically dependent on extensive test data [11, 14]. 12 Figure 8: Three-axis accelerometer [14]. Although heralded as the best tire sensing technique of the three studied, this conclusion was largely biased by the size, weight, and cost advantage of the accelerometer. It was even noted in the report that the optical technique produced the best results and held the most potential [11]. Matsuzaki et al. [15] developed strain sensors using the capacitance change of interdigital metal fingers. In one version, copper film attached to a polyimide substrate was etched accordingly and a flexible epoxy resin filled the voids between the fingers, in which a photo of said sensor is shown in Figure 9. However, it was noted that the polyimide and epoxy resin were still stiffer than the underlying rubber, which decreases the accuracy of the data and sensor robustness [15]. 13 Figure 9: Flexible epoxy strain sensor developed for tire use [15]. Improving upon this concept, the authors employed photolithography to take advantage of natural rubber as the substrate, as shown in Figure 10. The sensor was able to capture much more deformation and was easier to fabricate [16]. 14 Figure 10: Tire strain sensor using natural rubber as the substrate [16]. Matsuzaki et al. [17, 18] also investigated using the wires of the steel belt beneath the tread to measure capacitance changes between them as a consequence of their movement during deformation. The steel wire capacitor was connected with a fixed resistor to form a capacitance-resistance oscillator circuit in which a tuning frequency and peak power spectrum were derived from the received signal wave [17, 18]. Although utilizing only the tire itself as the sensing element, implementation is difficult given that the steel wire belt is sandwiched between other layers and encased in rubber. Figure 11 conveys this issue. 15 Figure 11: Using steel wires of a tire to detect deformation through capacitance changes [17]. Sergio et al. [19] studied an improvement upon the previous approach in which the rubber was treated as a resistive element and cross coupling of facing steel wires was taken into account [19]. Hiraoka et al. [20] first conceived the idea of measuring tire deformation using a camera attached to the rim. Using the well established metrology approach known as digital image correlation, the authors were able to measure displacement in the lateral and longitudinal directions over a given area of the contact patch, making this the first full- field sensing approach. A laser spotlight projection method was employed to determine 16 the radial displacement and subtract its effects on the in-plane displacements. This method required a separate image be taken using just the laser diodes [20]. Figure 12 below shows their camera and accompanying laser diodes. Figure 12: Camera and laser diodes used to capture in-plane and out-of-plane displacements [20]. A second study sought to eliminate the need for the second image by measuring the deformation at the top of rubber blocks glued to the inner liner, as shown in Figure 13, as well as the deformation of the surrounding inner liner. Through several assumptions, namely that strain at the top of the blocks was negligible and assumed to be zero, they purportedly determined the displacement in all three directions, including necessary corrections to the in-plane displacements [21]. However, their approaches were sub- optimal due to several reasons including a coarse sub-pixel interpolation scheme, not 17 calculating deformation gradients, and using a low resolution camera. Figure 13: Diagram of camera imaging the tire surface with rubber blocks attached [21]. 2.4 Motivation Of the techniques and methods reviewed, the contacting ones inherently introduce errors by interfering with the sensed surface. On the other hand, the non-contacting designs also exhibit uncompromising disadvantages. Magori et al.?s [9] ultrasonic sensor is greatly susceptible to interfering signals bouncing within the tire cavity. Hiraoka et al.?s [20] camera and laser diode method required two images be captured in succession, which would never provide relevant information with a rotating wheel. Although termed 18 non-contacting, Matsuzaki et al.?s [21] improvement upon the laser diodes still interferes with the sensed surface through the gluing of rubber blocks, and the assumptions made were questionable in obtaining truly full-field data from the contact patch. Electronic stability control systems demand certain tire parameters in order to infer their state in a vehicle dynamics scenario, such as tire friction coefficients and slip angles. However, little research has been done to derive accurate parameters from direct sensing output [22-24]. It is suspected that the lack of an accurate technique to measure the purest form of contact patch information, deformation, has hindered this effort. Thus, the goal of this research was to develop an accurate, full-field, and non-contact sensing technique for measuring tire contact patch deformation and demonstrate its ability to also reveal strain information similar to other studies. The full-field approach should provide a foundation upon which to identify key points in the contact patch to extract necessary parameters for stability control routines, as well as providing crucial information for future tire modeling and development. Similar to the work of Hiraoka et al. and Matsuzki et al. [20, 21], the technology utilized here employed a single area-scan white light digital camera to capture images of a deforming speckle pattern on the tire inner liner. However, unlike the aforementioned studies, a fine sub-pixel interpolation scheme was used, deformation gradients were calculated, a high resolution camera was utilized, and a novel method was included to calculate out-of-plane displacements. These improvements will be clarified over the course of the ensuing chapters. 19 Chapter 3: Theory 3.1 Proposed Technique Measuring tire deformation lends itself to the optical metrology technique commonly referred to as digital image correlation (DIC). DIC has been performed using many variants of a correlation function, correlating a reference image to a deformed one, that when minimized yields parameters describing a mapping that deforms the reference image to match the deformed one. The pioneering credit of this technique is often given to the researchers at the University of South Carolina for their work in the 1980?s [25- 27]; however, the distinction between digital image correlation and image tracking in general is related more to semantics rather than mathematical or algorithmic differences. For this reason, the explanations that follow will borrow heavily from the computer vision camp?s terminology. The central idea is to decompose complex global deformation into localized deformations for several sub-images, or subsets, of the original that can be approximated with lower order models. The process of tracking an image fundamentally requires calculation of a transformation, or deformation, mapping. For this research, the affine transformation was selected. 3.2 Warping Function The warping function is merely a transformation mapping reference pixel coordinates to deformed ones. Several transformations exist; however, the affine 20 transformation was employed in this study because of its relationship to physical deformation. This transformation has six parameters that correspond to translation, rotation, and scale. The following equation conveys the correspondence between reference and deformed coordinates. ? x;p = ?? ?? = 1 + ?1 ?3 ?5 ?2 1 + ?4 ?6 ? ? , (1) where p1 and p4 comprise scale, p2 and p3 comprise rotation, p5 and p6 are translation, and x? and y? are the deformed coordinates. The warping occurs with respect to a reference point. For this work, the subset centers were used so that the entire subset was assumed to experience the deformations calculated. Using conventional deformation notation in Cartesian coordinates, the physical significance of the affine parameters are given in Table 1. It becomes clear that the affine warp entails displacement and its gradients. It should be noted that these parameters correspond to in-plane deformation. These parameters are pictorially depicted in Figure 14. Table 1: Physical meaning of affine parameters Affine Parameters p1 p2 p3 p4 p5 p6 Description scale rotation rotation scale translation translation Deformation Notation ?? ?? ?? ?? ?? ?? ?? ?? u v 21 Figure 14: Affine transformed subset. 3.3 Image Tracking 3.3.1 Preliminaries Image tracking begins by first specifying a template, or reference, image. It should entail a unique pattern so that it may be distinguished from its surroundings. A random speckle pattern was chosen to provide a locally unique pattern for any given area [28]. The proposed technique is able to return a dense array of deformation information approximating the imposed global deformation by segmenting a single speckled image into a grid of subsets, which may or may not overlap. The second image and those thereafter are the deformed images. Example speckle 22 images depicting a reference and deformed state are shown in Figure 15. Figure 15: Example speckle images. Panel A is the template image and Panel B is the deformed image. A multitude of tracking algorithms exists, each of which attempt to identify the deformed image that corresponds to the template image through a warping function. It is this function that contains the parameters descriptive of the underlying deformation. The deformation is assumed to be sufficiently small so that necessary mathematical expansions may be truncated to their first order approximations. Even if overall deformations are large, one could simply acquire more images during the deformation process so that deformations from image to image still remain small. The basic tracking algorithms used in this work are termed the forward and inverse compositional update algorithms [29]. 3.3.2 Subset Size Selection Observing that the original template image must be segmented, an obvious question asks what subset size to use? Pan et al. [30] dealt with this question by using a 23 simple algorithm that checks and increases the subset size according to the sum of the squares of the template subset gradients. This approach is based upon the variance in displacement that was described in detail by Sutton et al. [31]. The following equation shows the inverse proportionality of the sum of squares gradients to the variance. ??? p ? 2?2 ?? 2x , (2) where Var(p) is the variance in deformation parameters, ? denotes standard deviation, and ?T is the template image gradient in the x and y directions. Since one would want the variance to be small, the sum of the squared gradients should be maximized. Pan et al. [30] proposed increasing the subset size from 11x11 pixels incrementally by two pixels until both gradient terms exceed a set threshold. This technique was utilized in this work. 3.3.3 Forward Compositional Algorithm The forward compositional algorithm [29] is a slight modification to the forward additive algorithm that was first described and generalized by B. D. Lucas and T. Kanade [32]. The goal of the algorithm is to minimize the error between the template and deformed images, thereby calculating estimates of the parameters composing the specified transformation. The Euclidean norm is used for the error function where the deformed image is transformed by some warping function. If the parameters of the warping function perfectly describe the transformation, then the error will equal zero. ????? = ? ? ? x;?p ; p ? ? x 2, x (3) where x is the vector of coordinates [x, y]T, p is the vector is deformation parameters, I is 24 the deformed image, T is the template image, and W(x;p) is the warping function. Since the deformation parameters are unknown, the error function must be expanded so that they may be iteratively solved within some tolerance. The incremental updates are denoted ?p and composed with the current estimate of p. Expanding I(W(x;p)) about ?p using Taylor?s series yields equation 4. ????? = ? ? x; p + ?? ? x; p ? ???p ? ?p ? ? x 2 x (4) Equation 4 now provides the means to solve for the incremental updates to p. To do so, the partial derivative of the error function with respect to ?p is set equal to zero and rearranged as follows. ?????? ??p = 0 = 2 ?? ? x; p ? ?? ?p ? ? ? x; p + ?? ? x; p x ? ???p ? ?p ? ? x (5) ?p ?? ? x; p ? ???p ? ?? ? x; p ? ???p x = ?? ? x; p ? ???p ? ? ? x; p ? ? x x (6) ?p = ??1 ?? ? x; p ? ???p ? ? ? x; p ? ? x , x (7) where ? = ?? ? x; p ? ???p x ? ?? ? x; p ? ???p is denoted the Hessian matrix. This formulation can also be described as a Gauss-Newton iteration scheme applied to a nonlinear least squares problem. This is an approximate variant of a Newton iteration scheme in which the second order portion of the Hessian matrix is excluded so 25 that the Hessian is actually a Gauss-Newton approximation [33]. The condition to stop iterating is when the Euclidean norm of ?p is less than a tolerance specified by the user. For this work, a value of 0.001 appeared to be the limit, where once the Euclidean norm was less than this value, any smaller value did not appreciably improve the estimates. 3.3.4 Inverse Compositional Algorithm S. Baker and I. Matthews also developed an inverse version of the algorithm previously described in which the roles of the template and deformed images are switched [29]. The equations that follow elucidate the formulation of the inverse compositional iterative scheme as done with the forward compositional algorithm. ????? = ? ? x;?p ? ? ? x; p 2 x (8) ????? = ? x + ?? ? ???p ? ?p ? ? ? x; p 2 x (9) ?????? ??p = 0 = 2 ?? ? ?? ?p ? ? x + ?? ? ???p ??p ? ? ? x; p x (10) ?p? ?? ? ???p ? ?? ? ???p = ?? ? ???p ? ? x ? ? ? x; p xx (11) ?p=???1 ?? ? ???p ? ? x ? ? ? x; p x , (12) where ?? = ?? ? ???p ? ?? ? ???p x is denoted the Hessian matrix for the inverse algorithm. 26 3.3.5 Coarse Search and Initialization The iteration methods previously described demand a sufficiently close initial guess. This was achieved using a brute force search in a given square window around the subset of interest where the deformed subset was displaced in one pixel increments in the x and y directions. The deformed subset that gave the lowest error according to equation 3 corresponded to the initial integer estimates of p5 and p6. All other deformation parameters were initialized to zero. 3.3.6 Levenburg-Marquardt Iteration The Levenburg-Marquardt method was used to improve the global convergence character away from local minima. The basic concept is to increase or decrease the importance of the Gauss-Newton approximation to the Hessian matrix by including a diagonal approximation of the Gauss-Newton Hessian multiplied by a scale factor ?. The diagonal approximation allows larger step sizes when the current estimate is farther away from the minimum. Equations 13 and 14 shows the diagonal approximation addition for both the forward and inverse compositional algorithms, where the subscript ?LM? denotes ?Levenberg-Marquardt? [29]. ? = ?? ? x; p ? ???p x ? ?? ? x; p ? ???p + ? ? ???? ?? ? x; p ? ???p 2 x (13) ?? = ?? ? ???p ? ?? ? ???p x + ? ? ???? ?? ? ???p 2 x (14) The scale factor is initialized to 0.01 and divided by 10 if the error has decreased 27 and multiplied by 10 if it has increased. Furthermore, with an increase in error, the parameters are not updated since this ensures only those updates that decrease the error will be composed with the current estimate of p [29]. 3.3.7 Sub-pixel Interpolation Noting the warping function, it is clear that the deformed image pixels may reside at sub-pixel locations. This poses an issue since the original deformed image is composed of pixels with integer coordinates and the intensities at the sub-pixel locations must be retrieved during the iteration steps. Sub-pixel interpolation attempts to alleviate this; however, a wide array of interpolation functions exist. Bicubic spline interpolation was chosen because it maintains the intensities at integer locations and provides a continuous two dimensional representation of the intensity surface [34]. This is in contrast to the coarser bilinear interpolation used in Hiraoka et al.?s work [20, 21]. Equation 15 describes the bicubic interpolation scheme. ? ??,?? = ? ?,? ? ?? ??0 ? ? (?? ? ?0)? , 3 ?=0 3 ?=0 (15) where C(i,j) are the sixteen spline coefficients and (x0, y0) are the truncated integer coordinates corresponding to (x?, y?). Any sub-pixel coordinate is located within an area bounded by four pixels at integer coordinates, as demonstrated in Figure 16. The sixteen spline coefficients require sixteen equations that are easily formed using the four corner pixels and eight pixels surrounding those. The set of equations are stated in the Appendix. 28 Figure 16: Sub-pixel demonstration. 3.4 Illumination Scale and Offset Illumination of the imaged surface may vary from the reference to the deformed state as a result of out-of-plane movement or varying current to the lighting. In this study, variations in illumination were assumed linear for each subset and thus accounted for with a scale and offset constant that were iteratively estimated along with the affine parameters. These variables are shown in equation 16 influencing the template image. ?(x)? = ? ? ? x + ?, (16) where T(x)i is the template image adjusted for illumination variation, ? is the scale, and ? is the offset. 29 These parameters are initialized to one and zero, respectively. Typically, these variations in illumination are removed through a normalization in which the mean is subtracted and standard deviation divided from the respective template and deformed images. However, that approach was avoided in this research due to the inclusion of blur estimation that creates a dependence on an unknown number of pixels surrounding those of a given subset. In other words, the normalization assumes the mean and standard deviation are functions of only the pixels within a given subset which is not the case in the event of blurring as described in the next section. 3.5 Blur Estimation In a monocular vision setup, a standard lens has a specific plane in which the imaged surface is sharply focused. Should the surface move away from or toward the lens, a blurring effect is induced that can be described using geometrical optics. Blurring can be approximated by convolving each pixel of a focused image with a radially symmetric two dimensional Gaussian function. Since images are discrete data arrays, the normalized and discrete two dimensional Gaussian is given in the following equation. ? x;? = exp x Tx ?2?2 exp xTx ?2?2 r y?r ? x?r , (17) where r is the radius of blur and ? is the standard deviation of the Gaussian distribution. The pixel weights of the Gaussian are governed by ? which can be directly related to a radius of blur according to the following equation. ? = 3? (18) 30 The reasoning for this equation is a result of the fact that Gaussian weights are practically zero beyond a distance of 3?. Figure 7 displays a three dimensional representation of a two dimensional Gaussian function using a ? of 2. Figure 17: Two dimensional Gaussian example at ? = 2. The convolution operation was performed on the template image since only the deformed image can experience blurring from out-of-plane tire displacement. The contact patch can only move inward, toward the camera, thus ? was forced to be greater than or equal to zero; however, the zero case is described by taking the limit of the Gaussian function as it approaches zero, as shown in the following equation. lim ??0 ? x;? = lim ??0 1 2??2 ? ?12?? 2+?2 ?2 = ?(?, ?) (19) 31 The result is the dirac-delta function, which when convolved with an image about any pixel in the discrete domain simply yields the original pixel. In the discrete domain, convolution of a template subset with the two dimensional Gaussian is given in equation 20. ?(x) ? ?(x;?) = ?(x) ? ? ??? ? ??? ?(x;?) (20) T(x) was replaced with ?(x) ? ?(x;?) in equation 3 and expanded about ?, where ? = ?2, using Taylor series, as shown in equation 21. ????? = ? ? x; p + ?? ? x; p ? ?? ?p ?? p ? ? x ? ? x;? x ? ? x ? ?? x;? ?? ??? 2 (21) The notion of calculating deformation and blur parameters simultaneously must be credited to the work of Myles and da Vitoria Lobo [35]; however, their work initially blurred both images to smooth the data and their iterative approach was different from the one used in this research. The next section describes how illumination variation and blurring were included in the original forward compositional approach in addition to using the inverse compositional approach for deriving initial estimates. 3.6 Complete Algorithm The algorithm used for this work has three main stages: coarse search, inverse compositional update, and forward compositional update with illumination and blur accommodation. The coarse search, as described in section 3.3.5, provided the initial guess for the translational parameters. The second stage was included to increase 32 computational efficiency by providing estimates for all six deformation parameters. In fact, blur estimation was included in the inverse compositional algorithm, but the computation was far slower than the forward approach. The third stage includes estimation of the illumination variation and blur parameters. The Taylor expansion of equation 3 with respect to all the estimated parameters is given in equation 22 in which all second order terms were omitted. ????? = ? ? x; p + ?? ? x; p ? ?? ?p ?? p ? ? ? ? x ? ? x;? x ? ? ?? x ? ?? x;? ?? ??? ?? x ? ? x;? ??? ? ? ??? 2 (22) ? is actually included with the Gaussian convolution of the template image; however, because ? is an added constant, the convolution with the two dimensional Gaussian is just the constant itself. An improvement was noted in the calculation of the partial derivative of the Gaussian with respect to ? that takes advantage of the solution to the heat equation in which ?? x;? ?? = 2?2?(x;?). Since the Laplacian operator is linear, it was moved to the template image so that it would only have to be computed once for the entire image, resulting in the revised expansion. ????? = ? ? x; p + ?? ? x; p ? ?? ?p ?? p ?? ? ? x ? ? x;? x ? ? ???2? x ? ? x; ? ??? ?? x ? ? x;? ??? ? ? ??? 2 (23) The following two figures illustrate the complete algorithm in a flowchart. It should be noted that this algorithm and all supporting functions were coded using MATLAB? [36]. 33 Figure 18: Complete algorithm flowchart. The coarse search and inverse update algorithm are presented. 34 Figure 19: Complete algorithm flowchart continued. The forward update algorithm is presented. 35 Chapter 4: Simulation and Results Testing the algorithm presented in Chapter 3 required simulating speckle image deformation. Following the work of Zhou and Goodson [37] and Pan et al. [34], template and deformed speckle images were created using the sum of Gaussian equations that follow. ?? ?, ? = ?? ? ??? ? ? ? ?? 2 + ? ? ?? 2 ?2 ? ?=1 and (24) ??(?,?) = ?? ? ??? ? ? ? ? ? ? ? ???? ?? ? ???? ? ? ? ?? 2? ?=1 + ? ? ? ? ? ? ???? ? ? ? ???? ? ? ? ?? 2 /?2 , (25) where It(x, y) and Id(x, y) are the template and deformed images respectively, s is the number of speckles, Ak is the random vector of amplitudes for each speckle, (xk, yk) are the random speckle coordinates, and R is the speckle radius. For an example of these images, refer to Figure 15. As shown in equation 25, the input deformation parameters are applied to the entire image; however, the image was divided into overlapping subsets in which the parameters were calculated for each. The overlap simply ensures that the perimeter subsets are not too close to the image boundaries. Scale and offset were easily added using equation 20, and blurring was accomplished using equation 19. Also, it should be noted that the image coordinates 36 place the origin in the center; therefore, odd image dimensions were used, specifically 257 pixels for the following simulations. For all the following simulations, s was set to 1400 and R to three pixels. A 5x5 grid of subsets was used to collect the deformation and radius of blur parameters. The subset size was changed according to the subset size selection routine. Figure 20-23 display the deformed image overlaid with the displacement vectors. Furthermore, the tables display the parameter values for five subsets corresponding to the corners and center in which the subset coordinates are read in the conventional x and y directions. In the first simulation, p5 and p6 were set to 13.5 pixels each and a subset size of 41x41 pixels was needed. An overlap of 10 pixels was also used. This simulation was chosen to demonstrate the effect of translation to a sub-pixel location in which all the displacement gradients should be approximately equal to zero. 37 Figure 20: Simulation result. p5 = 13.5px, p6 = 13.5px. 38 Table 2: First simulation subset values. Subset p1 (x10-4) p2 (x10-4) p3 (x10-4) p4 (x10-4) p5 p6 ? (1,1) -3.3709 0.3884 -0.8251 0.2823 13.5006 13.4981 0.4080 (1,5) -1.9292 -1.4894 -1.1539 2.4973 13.5010 13.5008 0.3996 (3,3) -1.6146 0.7335 0.4244 0.8759 13.4973 13.4979 0.3809 (5,1) 0.1649 2.1702 -1.8909 -1.8585 13.4978 13.5011 0.3931 (5,5) -0.0853 -0.3874 -0.5208 -0.0772 13.5028 13.4992 0.3956 Average Percent Error - - - - 0.0007% 0.0058% - In the second simulation, p1 and p4 were each set to 0.08, corresponding to an increased magnification effect. The necessary subset size was 41x41 pixels and 10 pixel overlap was used. The following figure illustrates the effect of magnification on the subset translation. This simulation was chosen to demonstrate the effect of scale. 39 Figure 21: Simulation results. ?1 = 0.08 and ?4 = 0.08. 40 Table 3: Second simulation subset values. Subset p1 p2 (x10-3) p3 (x10-3) p4 p5 p6 ? (1,1) 0.0869 -0.1975 0.0068 0.0870 -5.4777 -5.4770 0.3782 (1,5) 0.0865 0.1403 0.2759 0.0871 -5.4769 5.3033 0.3655 (3,3) 0.0868 -0.3467 -0.1450 0.0866 -0.0828 -0.0857 0.3799 (5,1) 0.0867 0.2968 0.2110 0.0870 5.3037 -5.4806 0.4064 (5,5) 0.0870 0.1271 0.1679 0.0870 5.3024 5.3012 0.3451 Average Percent Error 8.475% - - 8.675% - - - In the third simulation, p2 and p3 were set equal to 0.2 and -0.2, respectively to induce a rotation. Thus, this simulation demonstrates the algorithm?s ability to detect rotation. The subset size was set to 41x41 pixels with a 10 pixel overlap. The results follow. 41 Figure 22: Third simulation plot. ?2 = 0.2 and ?3 = ?0.2. 42 Table 4: Third simulation subset values. Subset p1 p2 p3 p4 p5 p6 ? (1,1) -0.0384 -0.1925 0.1922 -0.0385 -9.6904 14.5409 0.3704 (1,5) -0.0385 -0.1925 0.1924 -0.0383 14.1542 9.7693 0.3733 (3,3) -0.0388 -0.1923 0.1924 -0.0384 -0.1565 0.2295 0.3779 (5,1) -0.0382 -0.1925 0.1921 -0.0384 -14.4590 -9.3073 0.3861 (5,5) -0.0385 -0.1924 0.1926 -0.0387 9.3869 -14.0780 0.3736 Average Percent Error - 3.78% 3.83% - - - - The fourth simulation mimics an angled shearing effect along with blurring in which: ?1 = 0, ?2 = 0.2, ?3 = 0.2 , ?4 = 0, and ? = 2. The subset size was set to 41x41 pixels with a 10 pixel overlap. 43 Figure 23: Fourth simulation plot. ?2 = 0.2 , ?3 = 0.2, and ? = 2. Table 5: Fourth simulation subset values. Subset p1 p2 p3 p4 p5 p6 ? (1,1) 0.0402 0.2034 0.2118 0.0433 -12.9829 -12.9808 2.0088 (1,5) 0.0383 0.2052 0.2107 0.0411 12.8006 -7.8396 2.0243 (3,3) 0.0411 0.2036 0.2057 0.0423 2.5125 2.4985 2.0218 (5,1) 0.0413 0.2064 0.2086 0.0405 -7.8237 12.8382 2.0318 (5,5) 0.0441 0.2079 0.2021 0.0449 17.9852 18.0424 2.0014 Average Percent Error - 2.65% 3.89% - - - 0.8810% 44 As one can see from the tabular results, the calculated estimates are in good agreement with the imposed deformation with the average percent errors corresponding to the input deformations being relatively small. In the absence of blurring, the algorithm yields a parameter value equal to approximately 0.4. This is likely a result of the Gaussian function used, in which 0.4 or less can be approximated as essentially zero. These simulations validate the ability of the algorithm to detect all six affine parameters, even in the presence of blurring. Errors are also introduced as a result of the interpolation function. More accurate functions are available but the computational expense increases severely. It was also possible that better estimates could have been attained had more iterations been performed; however, more iterations only decreases computational efficiency. This compromise between accuracy and efficiency was beyond the scope of this research but should be pursued in future work. 45 Chapter 5: Experimental Setup 5.1 Rim and Tire Constraints Before a camera and lens could be selected, certain dimensional constraints from the rim and tire had to be measured. These influence parameters such as the lens focal length, camera height, and lens height among others. For this study, a standard reference tire was used that fit a 16? rim. A 16? three piece rim made by BBS Wheels was chosen to ease and protect installation of the electronics, as shown in Figure 24. 46 Figure 24: Three piece rim. Figure 25 displays the important dimensional constraints imposed. 47 Figure 25: Rim and tire dimensional constraints. With these measures, it became apparent that the camera would have to fit within the 50mm of the hoop basin and preferably be no more than 35mm in height. Furthermore, the 105mm distance from the edge of the bead flange to the tire inner liner (measured to the middle since some curvature is present) was the limiting factor in choosing an appropriate focal length. Also, the desired field of view (FOV) was set to 100x100mm to include as much of the contact patch as possible. 5.2 Camera To select a sufficient camera, first a desired pixel resolution must be determined. 48 The following equation provided a good working number of 1000x1000pixels. ??? = ??? ?0 ?? , (26) where res is the pixel resolution, d0 is the displacement accuracy desired (set to 1x10-6m), and dI is the image displacement accuracy (set to 0.01 pixels). The Ueye 1466 LE S- mount board camera made by IDS Imaging Development Systems GmbH was ultimately chosen based on the various specifications shown in Table 6. Figure 26 shows a photograph of the camera. Table 6: IDS Ueye 1466 LE camera specifications. Desired Specifications Actual Specifications More than 1000x1000px resolution 2048x1536px Small form factor (less than 50x50mm) 36x36mm board size Less than 35mm height Approximately 23mm Simple computer interface USB interface with LabVIEW? drivers Easy to mount Four corner screw holes 49 Figure 26: IDS Ueye 1466LE camera. 5.3 Lens The necessary focal length was calculated using the following equation. ? = ? ? ??? ??? + ??? , (27) where f is the focal length, w is the working distance, sfd is the shortest sensor format dimension, and FOV is the field of view. The working distance was 105mm as shown in Figure 25 and the shortest sensor format dimension was 3.6mm. Thus, the focal length was calculated to be 4.58mm. A Lensation low distortion, 3.5mm focal length lens was ultimately chosen of which the reason is explained in the following paragraph. Figure 27 displays the lens used. 50 Figure 27: Lensation S-mount lens. The lens chosen has a 3.5mm focal length and made to have a low distortion. It should be noted that the focal length was rated for a minimum working distance of 200mm. Because the working distance within the tire is only 105mm, the distance from the camera sensor to the back of the lens must be increased. Doing so in turn decreases the available field of view, hence the reason a smaller focal length lens was employed. This process can be described by equations, but it requires advanced knowledge of the lens components and their construction, which was not made available. Therefore, the minimum working distance of the lens was adjusted by visual inspection, or until an imaged surface at 105mm appeared reasonably focused. Although this approach seems imprecise, the algorithm presented in Chapter 3 seeks to calculate changes from a reference state so that a perfectly focused template image is not necessary. Furthermore, a smaller focal length could conceivably achieve a larger FOV; however, there are physical limits to the resolution that can be afforded. Without sophisticated lens technology, only fisheye lenses can achieve large FOV?s at small focal 51 lengths but these induce a significant barrel distortion that must be corrected before any further image processing is performed, which inherently introduces errors. 5.4 Illumination In speckle metrology, uniform object illumination is preferable; however, this often involves expensive modules with large power requirements and large sizes with respect to the tire cavity being utilized here. Because illumination variation was modeled in the algorithm, precise uniformity in lighting was not deemed a significant constraint. Four-pin white light emitting diodes (LEDs) were chosen for their ease of mounting, high lumen output, low power requirement, and compact size. A row of five LEDs was placed on two opposing sides of the lens. The figure below shows one of these LEDs. Figure 28: Four-pin white light LED. 52 The LEDs were wired in parallel with each having a series resistor to limit the current being applied through a power supply. 5.5 Mounting Affixing a camera inside a tire cavity posed some challenges. First, a mounting plate was fabricated from 6061 aluminum 3/8? flat bar stock that would serve as the base to which all other parts would be attached. Holes were drilled in the top face to accommodate the lens and allow four screws to secure the camera to the plate at its four corner mounting holes. Cutouts were milled from opposing sides, allowing space for the LEDs to occupy. Threaded holes were made in two opposing sides so that flanges could be fastened. The flanges were cut from 5052 aluminum sheet and bent as needed to accommodate the curvature of the rim hoop. The flanges were mounted to the rim with screws to ensure a secure fit. The following figure displays a computer assisted design (CAD) assembly drawing of the mounting plate, flanges, and camera and lens. All parts were machined or formed in the Design and Manufacturing Lab that houses the on- campus machine shop. The final product is shown in Figure 30. 53 Figure 29: CAD assembly drawing. 54 Figure 30: Final assembly. Five LEDs were soldered to prototyping board and fastened as shown. The lens and camera are precisely positioned in the center of the mounting plate. Accessing an external connection to the electronics within the tire cavity was accomplished using a hermetically sealed plug from Pave Technologies in which an appropriately sized hole was drilled into one of the rim halves and a rubber gasket and jam nut provided an air tight seal. This plug is shown in Figure 31. 55 Figure 31: Pave Technologies hermetically sealed plug. 5.6 Pressure and Temperature Monitoring The tire pressure and temperature were also recorded for the sake of control and experimental repeatability. This was accomplished using the TPMS evaluation kit from Freescale Semiconductors which was chosen for its small form factor and ease of use. 5.7 Tire Surface Preparation A random speckle pattern was easily created on the tire inner liner using the overspray from a white spray paint. Since the tire surface was already a matte black, only the white paint was needed after first wiping down the liner to remove any residual dirt and oils. The following figure displays the speckled tire surface. 56 Figure 32: Speckled inner liner. 5.8 Tire Loading The tire was loaded vertically in compression using an MTS Q-Test? machine. An incremental displacement of 2mm was applied until 20mm was achieved. This protocol was similar to that in the work of Matsuzaki et al. [21]. The following photograph shows the tire in said machine. 57 Figure 33: Tire in MTS Q-Test machine. Note the aluminum head used to ensure load was evenly applied over the entire contact patch area. The loading conditions should not be considered equivalent to true vertical loading since the tire was being compressed on both sides instead of at the wheel hub. Future testing should make use of a proper tire testing machine such as the Flat-Trac III? made by MTS. 5.9 In-plane Deformation Correction The out-of-plane displacement causing blurring also corrupts the in-plane deformation due to magnification. This only influences the translation and scale parameters. Ideally, the relation between out-of-plane displacement and ? would be 58 exactly known from the thin lens equation for light rays; however, the lens used in this work was a multi-element lens that is governed by a different set of equations that could not be recovered from the manufacturer. In spite of this setback, the influence of out-of- plane displacement could be roughly quantified by empirically determining the blur and displacement relation through a careful calibration procedure described below. A flat upright surface was coated in speckles and spaced 105mm from the front of the lens as shown in Figure 34. The surface was then moved toward the camera in 2.5mm increments and an image was captured for each. This process was continued until 30mm had been covered. The images were then feed into the algorithm presented in Chapter 3. Unfortunately, the lens exhibited astigmatism in that blurring occurred non-uniformly across the FOV. Since much care was taken in aligning the camera, this aberration could only have been the result of a manufacturing defect. Figure 35 is a surface plot showing calculated blur parameters for ten 31x31 pixel subsets with no overlap. 59 Figure 34: Blur to displacement calibration procedure. Note the 2.5mm spaced lines between the camera and speckled surface. 60 Figure 35: Blur parameter surface plot. This plot demonstrates the astigmatism inherent to the lens used. To cope with this defect, the FOV was assumed to be planar in the tire contact patch and only normal loading would be applied so that the FOV would remain planar with increasing out-of-plane displacement. It was thought that calculating the blur parameter for a center subset and using that parameter to represent the blurring across the FOV would provide a reasonable approximation. However, as noted in Figure 36 the scale to blur relationship appears quadratic in nature, which is different from the linear expectation. 61 Figure 36: Scale to blur relationship for center subset. Concerned that the lens blurring effect was nonlinear, defining a relation between scale and blur was abandoned, and the simple assumption was made that the ratio of out- of-plane displacement to the distance between the lens and tire surface in the un- deformed state is directly proportional to the scale imposed by said displacement. Figure 37 demonstrates this assumption in which an object at an initial depth Z displaced toward the lens by ?Z appears to grow larger. The coordinate (x, y) represents a point on the object on the sensor plane and (x*, y*) is the displaced coordinate as a result of the out- of-plane displacement. The sensor plane to back of lens distance is denoted L; however, this quantity was not known in this research. 62 Figure 37: Proportionality of out-of-plane displacement to initial depth ratio to scale factor. Sutton et al. [38] used the thin lens equation to quantify the effects of out-of-plane displacement on in-plane deformation of a planar surface being imaged. The effect on displacements u and v is given equations 28 and 29. ?? = ? ? ? ? + ?? ? ? ? = ?? ? ? (28) ?? = ? ? ? ? + ?? ? ? ? = ?? ? ? (29) Essentially, the out-of-plane displacement scales the original image larger thereby creating the appearance that points are moving radially outward from the center so that u* and v* are apparent displacements rather than actual ones. It should be noted that the formulations above assume the image center is the origin at (0,0). As described earlier, out-of-plane displacement also affects the scale parameters. The following equations were also derived in Sutton et al.?s [38] paper and describe this effect. 63 ??? ?? ? ?? ? and (30) ??? ?? ? ?? ? , (31) where ????? and ????? are the apparent displacement gradients. With this, the determination of the scale correction is no longer calculated internally from the processed image data. A lens without astigmatism would alleviate the problems described. Since only normal loading would be applied, the out-of-plane displacement would be known, and the scale correction would be easily calculated and subtracted from the raw results. Although the lens defect made internal determination of out-of-pane displacement from blur impossible for this work, it seems relevant to describe the process that would have been used had the defect not been present. Without astigmatism, the blur parameters would have been generally equivalent over the FOV within some tolerance of error. The parameters could be calculated across the FOV of the calibration images mentioned previously and averaged to attain a representative parameter at that particular displacement. A function could then be fitted to the blur parameters versus averaged scale parameters. Then, the blur parameter calculated at any subset in a real experiment could be used to find the appropriate scale correction factor. Furthermore, a function could be fit to the blur parameter versus out-of-plane displacement so that blur parameters in experimental results could be related. The correction would be applied to each subset individually so that even a cambered contact patch could be detected. This procedure is outlined in the flowchart of Figure 38. 64 Figure 38: Blur to scale correction flowchart. This procedure would be used if the lens had no astigmatism. During this process, it was noted that speckle size contributed to convergence of the algorithm in that a dense pattern of small speckles is more difficult to track as blurring increases. For this particular setup, displacements greater than 15mm seemed to indicate significant blurring. This is an obvious result of the blurring process since it homogenizes pixel intensities and thereby removes fine details that may have delineated one subset from another. Therefore, a certain balance exists between creating small enough speckles to produce uniquely different subset patterns, but not so small that significant blurring makes the deformed subset patterns indistinguishable from one another. There is also a limit to the amount of blurring that can be detected before the 65 deformed image has lost too much information. These details should be explored in future work. 66 Chapter 6: Experimental Results and Analysis The output of the aforementioned algorithm includes a large amount of calculated data. The raw displacement data was plotted for each incremental displacement in addition to contour plots of the corrected scale parameters ???? and ???? for each subset center. Using the infinitesimal strain assumption [39], normal strains are approximately equal to their respective displacement gradients as shown in equations 32 and 33, assuming the imaged plane does not move out-of-plane. ? ?? = ?? ?? = ?1 (32) ? ?? = ?? ?? = ?4 (33) Furthermore, these strains were assumed to be the longitudinal and lateral strains in the tire contact patch after the appropriate correction had been made. For the experimental results, 37x37 subsets were used at a size of 41x41 pixels with a 25 pixel overlap. The results for six increments are shown corresponding to out-of- plane displacements of 2, 4, 6, 8, 10, and 12mm in which the load was recorded for each. Images at larger deflections were also recorded; however, the blurring induced beyond 12mm caused the iterations for some subsets to diverge from the correct parameters. This was attributed to the speckle pattern as described in section 5.9. Other than the raw displacement data, the coordinates were transformed such that the origin was at the image center and converted to units of millimeters using a 67 calibration target of a black and white checkerboard pattern in which each square was precisely 10x10mm. Simply dividing the 10mm by the number of pixels spanning a square?s length yielded a scale factor converting pixels to millimeters, which was 0.0990mm/px. The tire pressure and temperature were measured at 255kPa and 22?C. The pressure and temperature did not change over the course of the testing. The out-of-plane displacements are given in Table 7 along with the recorded load and the scale correction factor used in producing the strain results that follow. Table 7: Loads corresponding to applied out-of-plane displacements. Out-of-plane Displacement (mm) Load (N) Scale Correction 2 285.22 0.019 4 551.44 0.038 6 811.54 0.057 8 1071.54 0.076 10 1327.84 0.095 12 1580.00 0.114 The next several figures are arranged such that the raw in-plane displacement, shown as a contour plot with vectors overlaid, is presented first followed by contour plots of longitudinal and lateral strain. Negative strain indicates compression since a negative scale parameter shrinks the subset, and positive strain indicates tension. 68 At 2mm deflection, or vertical displacement, one would expect very little deformation. As expected, the raw displacement indicates little movement and the strain plots show negligible results. Figure 39: Raw pixel displacement at 2mm deflection. 69 Figure 40: Longitudinal strain at 2mm deflection. 70 Figure 41: Lateral strain at 2mm deflection. With 4mm deflection, the displacement understandably increases. The longitudinal strain begins to demonstrate a distributional trend in which more tensile strain occurs at the periphery of the center which shows less tensile strain. Lateral strain still appears negligible, although generally compressive. 71 Figure 42: Raw displacement at 4mm deflection. 72 Figure 43: Longitudinal strain at 4mm deflection. 73 Figure 44: Lateral strain at 4mm deflection. The longitudinal strain distribution remains with 6mm deflection but the values increase compared to that at 4mm. Lateral strain also increases, moving toward tension but still not appreciably. 74 Figure 45: Raw displacement at 6mm deflection. 75 Figure 46: Longitudinal strain at 6mm deflection. 76 Figure 47: Lateral strain at 6mm deflection. At 8mm, the strain evolution continues with increased tension in the contact patch but distributed as previously noted. 77 Figure 48: Raw displacement at 8mm deflection. 78 Figure 49: Longitudinal strain at 8mm deflection. 79 Figure 50: Lateral strain at 8mm deflection. With 10mm of deflection, the trend of increasing tensile strain continues as before. Furthermore, two bands at the lateral edges become more evident in the lateral strain plot in which more tension occurs in comparison to the rest of the contact patch. 80 Figure 51: Raw displacement at 10mm deflection. 81 Figure 52: Longitudinal strain at 10mm deflection. 82 Figure 53: Lateral strain at 10mm deflection. Finally, at 12mm deflection one sees the same strain distributions noted in the former plots. The longitudinal strain reached up to approximately 0.02mm/mm in tension in some places. The lateral strain increased but not nearly as much as the longitudinal strain. 83 Figure 54: Raw displacement at 12mm deflection. 84 Figure 55: Longitudinal strain at 12mm deflection. 85 Figure 56: Lateral strain at 12mm deflection. The raw displacement figures demonstrate that normal loading was achieved during the entire course of the compression test with displacements clearly increasing from one displacement to the next. The center of the contact patch shows reduced tensile strain at each out-of-plane displacement increment, while a ring surrounding the center shows a clear increase in tensile strain. Figure 57 plots the longitudinal strain along the lateral centerline for all 37 subset centers corresponding to those 37 subsets. Lateral strain appeared mostly negligible for all increments, as shown in Figure 58 which plots lateral strain profiles along the longitudinal centerline for all 37 subsets at each displacement increment. 86 Figure 57: Longitudinal strain profile at y = 0mm. 87 Figure 58: Lateral strain profile at x = 0mm. 88 Chapter 7: Discussion and Conclusions The evolution of strain over the vertical displacements reveals some interesting results. The distribution of tensile longitudinal strain does increase from one displacement to the next; however, the increase occurs mostly at the periphery of the center of the contact patch. This was very evident in Figure 57. The lateral strain plots indicate that compressive strain is dominant while tensile strain does increase at the lateral edges of the contact patch, although not markedly. The results lacked truly quantitative correctness due to the lens defect and out-of-plane assumptions; however, qualitative trends in strain could still be inferred. Previous studies supporting the trends found here are virtually non-existent since no one has used a sensitive, full-field technique to capture deformations in the contact patch. However, one paper by Matsuzaki et al. [23] does suggest the strain behavior noted here. They hypothesized that strain due to static friction is compressive and would thus subtract from the tensile strain induced from vertical loading. The authors plotted strain results from a finite element model over the length of the contact patch for several vertical loads ranging from 2500N to 4000N. The curves appear to dip slightly in the center as in Figure 57. However, the loading was significantly higher than that applied in this research, leaving any correlations inconclusive. 89 7.1 Contributions This thesis proved that digital image processing could successfully capture in- plane deformations in the presence of blurring and illumination variation under the assumptions given. It also proved that a monocular vision system could be placed inside a tire cavity to measure deformation in the contact patch. Although the lens defect made accurate and independent determination of out-of-plane displacement impossible, the blurring was still captured in the experimental images and strain maps were produced using the scale correction assumptions described. A procedure was shown for correcting the in-plane deformation due to out-of-plane displacement had the lens been non- defective. 7.2 Future Work This research should provide the foundation upon which numerous improvements are made to the hardware and software used. Much work is needed to determine an appropriate camera resolution to capture deformation in ranges critical to stability control systems. Incredibly high resolution cameras will detect very minute deformations but they may be impertinent to the desired outcomes of a real world intelligent tire. A finely tuned lens is also critical to developing an imaging system, especially in this work where the blurring effect should provide information for a third dimension. Furthermore, the lens could be adjusted to capture the entire contact patch, obviating any need to capture deformation over the period of contact duration. The algorithm presented attempted to preserve as much detail as possible; however, certain assumptions and approximations could be made to increase efficiency at 90 the expense of accuracy. This decision would have to be made after determining the necessary ranges of deformation, as mentioned previously. After careful design of the imaging system, an embedded architecture could be designed to process the image data within the tire instead of having to send it to an external computer. Ultimately, a wireless transceiver would be used to broadcast the relevant tire parameters extracted from the processed data to an electronic stability control system. 91 References [1] 2009, "AF093-210: Aircraft Tire Contact Patch Force and Shear Sensor," Airforce SBIR 09.3, US Department of Defense. [2] 2000, "Transportation Recall Enhancement, Accountability, and Documentation (TREAD) Act," 49, 106th Congress, United States. [3] NHSTA, 2006, "The Pneumatic Tire," US Department of Transportation. [4] 2011, "Rubber Manufacturers Association," http://www.rma.org/tire_safety/ tire_basics/what_is_inside_a_tire/?CFID=912052&CFTOKEN=49152535. [5] Erdogan, G., 2009, "New Sensors and Estimation Systems for the Measurement of Tire-Road Friction Coefficient and Tire Slip Variables," PhD Dissertation, University of Minnesota, Minnesota. [6] Brandt, M., Bachmann, V., Vogt, A., Fach, M., Mayer, K., Breuer, B., and Hartnagel, H. L., 1998, "Highly sensitive AlGaAs/GaAs position sensors for measurement of tyre tread deformation," Electronics Letters, 34(8), pp. 760-762. [7] Yilmazoglu, O., Brandt, M., Sigmund, J., Genc, E., and Hartnagel, H. L., 2001, "Integrated InAs/GaSb 3D magnetic field sensors for "the intelligent tire"," Sensors and Actuators a-Physical, 94(1-2), pp. 59-63. [8] Erdogan, G., Alexander, L., and Rajamani, R., 2010, "A novel wireless piezoelectric tire sensor for the estimation of slip angle," Measurement Science & Technology, 21(1). [9] Magori, V., Magori, V. R., and Seitz, N., "On-line determination of tyre deformation, a novel sensor principle," Proc. Ultrasonics Symposium, 1998. Proceedings., 1998 IEEE, pp. 485-488 vol.481. [10] Pohl, A., Steindl, R., and Reindl, L., 1999, "The "intelligent tire" utilizing passive SAW sensors - Measurement of tire friction," Ieee Transactions on Instrumentation and Measurement, 48(6), pp. 1041-1046. [11] 2001, "Intelligent tyre systems-state of the art and potential technologies," Technical Research Centre of Finland (VTT). [12] Tuononen, A. J., 2008, "Optical position detection to measure tyre carcass deflections," Vehicle System Dynamics, 46(6), pp. 473-483. 92 [13] Braghin, F., Brusarosco, M., Cheli, F., Cigada, A., Manzoni, S., and Mancosu, F., 2006, "Measurement of contact forces and patch features by means of accelerometers fixed inside the tire to improve future car active control," Vehicle System Dynamics, 44, pp. 3-13. [14] Audisio, G., Cheli, F., Melzi, S., and Velardocchia, M., "CyberTM Tyre for Vehicle Active Safety," Proc. XIX AIMETA, pp. 14-17. [15] Matsuzaki, R., and Todoroki, A., 2007, "Wireless flexible capacitive sensor based on ultra-flexible epoxy resin for strain measurement of automobile tires," Sensors and Actuators a-Physical, 140(1), pp. 32-42. [16] Matsuzaki, R., Keating, T., Todoroki, A., and Hiraoka, N., 2008, "Rubber-based strain sensor fabricated using photolithography for intelligent tires," Sensors and Actuators a-Physical, 148(1), pp. 1-9. [17] Matsuzaki, R., and Todoroki, A., 2005, "Passive wireless strain monitoring of tyres using capacitance and tuning frequency changes," Smart Materials & Structures, 14(4), pp. 561-568. [18] Matsuzaki, R., and Todoroki, A., 2006, "Passive wireless strain monitoring of actual tire using capacitance-resistance change and multiple spectral features," Sensors and Actuators a-Physical, 126(2), pp. 277-286. [19] Sergio, M., Manaresi, N., Tartagni, M., Canegallo, R., and Guerrieri, R., 2006, "On a road tire deformation measurement system using a capacitive-resistive sensor," Smart Materials & Structures, 15(6), pp. 1700-1706. [20] Hiraoka, N., Matsuzaki, R., and Todoroki, A., 2009, "Concurrent Monitoring of In- plane Strain and Out-of-plane Displacement of Tire Using Digital Image Correlation Method," Journal of Solid Mechanics and Materials Engineering, 3(11), pp. 1148-1159. [21] Matsuzaki, R., Hiraoka, N., Todoroki, A., and Mizutani, Y., 2010, "Optical 3D Deformation Measurement Utilizing Non-planar Surface for the Development of an "Intelligent Tire"," Journal of Solid Mechanics and Materials Engineering, 4(4), pp. 520- 532. [22] Matzukaki, R., Hiraoka, N., Todoroki, A., and Mizutani, Y., 2010, "Analysis of Applied Load Estimation Using Strain for Intelligent Tires," Journal of Solid Mechanics and Materials Engineering, 4(10), pp. 1496-1510. [23] Matsuzaki, R., Hiraoka, N., Todoroki, A., and Mizutani, Y., 2010, "Analysis of Applied Load Estimation Using Strain for Intelligent Tires," Journal of Solid Mechanics and Materials Engineering, 4(10), pp. 1496-1510. 93 [24] Matsuzaki, R., and Todoroki, A., 2008, "Wireless Monitoring of Automobile Tires for Intelligent Tires," Sensors, 8(12), pp. 8123-8138. [25] Chu, T. C., Ranson, W. F., Sutton, M. A., and Peters, W. H., 1985, "Appliations of Digital-Image-Correlation Techniques to Experimentail Mechanics," Experimental Mechanics, 25(3), pp. 232-244. [26] Bruck, H. A., McNeill, S. R., Sutton, M. A., and Peters, W. H., 1989, "Digital Image Correlation Using Newton-Raphson Method of Partial-Differential Correction," Experimental Mechanics, 29(3), pp. 261-267. [27] Pan, B., Qian, K. M., Xie, H. M., and Asundi, A., 2009, "Two-dimensional digital image correlation for in-plane displacement and strain measurement: a review," Measurement Science & Technology, 20(6). [28] Sutton, M. A., Orteu, J.-J., and Schreier, H. W., 2009, "5.1 Introduction to Image Matching," Image Correlation for Shape, Motion and Deformation Measurements: Basic Concepts, Theory and Applications, Springer, New York, pp. 81-83. [29] Baker, S., and Matthews, I., 2004, "Lucas-Kanade 20 years on: A unifying framework," International Journal of Computer Vision, 56(3), pp. 221-255. [30] Pan, B., Xie, H. M., Wang, Z. Y., and Qian, K. M., 2008, "Study on subset size selection in digital image correlation for speckle patterns," Optics Express, 16(10), pp. 7037-7048. [31] Sutton, M. A., Orteu, J.-J., and Schreier, H. W., 2009, "5.7 Statistical Error Analysis," Image Correlation for Shape, Motion and Deformation Measurements: Basic Concepts, Theory and Applications, Springer, New York, pp. 113-117. [32] Lucas, B. D., and Kanade, T., 1981, "An iterative image registration technique with an application to stereo vision," Proceedings of the 7th international joint conference on Artificial intelligence - Volume 2, Morgan Kaufmann Publishers Inc., Vancouver, BC, Canada. [33] Dennis, J., J. E., and Schnabel, R. B., 1983, "Nonlinear Least Squares," Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Englewoods Cliffs, New Jersey, pp. 218-228. [34] Pan, B., Xie, H. M., Xu, B. Q., and Dai, F. L., 2006, "Performance of sub-pixel registration algorithms in digital image correlation," Measurement Science & Technology, 17(6), pp. 1615-1621. [35] Myles, Z., and da Vitoria Lobo, N., 1998, "Recovering affine motion and defocus blur simultaneously," Pattern Analysis and Machine Intelligence, IEEE Transactions on, 20(6), pp. 652-658. 94 [36] 2010, "MATLAB," The Mathworks Inc., Natick, Massachusetts. [37] Zhou, P., and Goodson, K. E., 2001, "Subpixel displacement and deformation gradient measurement using digital image/speckle correlation (DISC)," Optical Engineering, 40(8), pp. 1613-1620. [38] Sutton, M. A., Yan, J. H., Tiwari, V., Schreier, H. W., and Orteu, J. J., 2008, "The effect of out-of-plane motion on 2D and 3D digital image correlation measurements," Optics and Lasers in Engineering, 46(10), pp. 746-757. [39] Reddy, J. N., 2008, An Introduction to Continuum Mechanis: with Applications, Cambridge University Press, NewYork. [40] Keys, R., 1981, "Cubic convolution interpolation for digital image processing," Acoustics, Speech and Signal Processing, IEEE Transactions on, 29(6), pp. 1153-1160. 95 Appendix Set of Equations for Bicubic Interpolation Equation 15 and its derivatives at the four surrounding pixels are used to obtain the set of equations that when solved yield the sixteen spline coefficients [40]. ? 0,0 = ? 0,0 = ?00 ? 1,0 = ? 1,0 = ?00 + ?10 + ?20 + ?30 ? 0,1 = ? 0,1 = ?00 + ?01 + ?02 + ?03 ? 1,1 = ? 1,1 = ??? 3 ?=0 3 ?=3 ?? 0,0 = ?? 0,0 = ?10 ?? 1,0 = ?? 1,0 = ?10 + 2?20 + 3?30 ?? 0,1 = ?? 0,1 = ?10 + ?11 + ?12 + ?13 ?? 1,1 = ?? 1,1 = ???? 3 ?=0 3 ?=1 ?? 0,0 = ?? 0,0 = ?01 ?? 1,0 = ?? 1,0 = ?01 + ?11 + ?21+?31 ?? 0,1 = ?? 0,1 = ?01 + 2?02 + 3?03 ?? 1,1 = ?? 1,1 = ???? 3 ?=1 3 ?=0 ??? 0,0 = ??? 0,0 = ?11 ??? 1,0 = ??? 1,0 = ?11 + 2?21 + 3?31 ??? 0,1 = ??? 0,1 = ?11 + 2?12 + 3?13 96 ??? 1,1 = ??? 1,1 = ????? 3 ?=1 3 ?=1