Link Copied
The accelerometers, tilt meters and GPS data that one collects, even if only at 8 second intervals or 2000 times per second, starts to create huge data series.
The first problem with these series is the thermal impact, this is not something one can ignore.
So a sample from a tilt meter:
Clearly there are two influences, a loss line and then thermal, the loss line is measured over decades and the temperature over a day, so temp dominates for a little while, but then the loss line kicks in.
We can do a linear regression in many ways, Fortran, EXCEL and C# as three samples. EXCEL gives the best graphs but it takes a lot of time, useful as you work out techniques. C# gives better graphs than Fortran for the same work, but Fortran is faster. C# does not handle non-linear easily. If you are doing a lot use Fortran or C#, personal choice.
If we do the linear regression in EXCEL we get
| Coefficients | Standard Error | t Stat | P-value | |
| Intercept | 4.579077353 | 0.000400117 | 11444.35258 | 0 |
| X Variable 1 | -1.73446E-07 | 8.35188E-10 | -207.6728992 | 0 |
The P value is not really zero, but such high t-STATS test the ability of real numbers to complete the analysis, anything over 2 in t-stat is the regression is solid and 200 is there is no argument.
The -1.7e-7 is not important till you realize there are 10000 steps per day, 3.65 million in a year and 365 million in a century, a bridge should last a century and we do not want the piers to tilt that far.
a residual plot and the trend is mostly gone and we get
The secondary analysis is to look at the residuals and here one can use FFT, but the answer is trivial at one day cycle and the better method is to use polynomial or such regression, but here I use the linear to take out the temperature dependence.
| Coefficients | Standard Error | t Stat | P-value | |
| Intercept | 4.528786381 | 0.000115415 | 39239.23 | 0 |
| X Variable 1 | -0.001623634 | 5.61804E-06 | -289.004 | 0 |
and the residual plot is
Now I am stuck, the only real solution is Fortran - maybe C++ if you are a masochist and we are now looking at lots of records, where the difference is the measured acceleration - thermal, traffic or construction. If we tease apart the data we will find the polynomial elements that relate to traffic and construction and the base one is the one left over - minor thermal. In this case because we did a few months without traffic we know the lowest curve is no loads other than natural, bit of wind, river force etc..
So we use Fortran to subset the data into nice groups and then apply polynomial type regression, ie I found FITPACK and now I have to see if it will work.
One tests out the methods with EXCEL and then you code them.
The interesting issue is the use of the Central Limit Theorem on very large data sets.
Can you make two new charts, TiltX and X Variable 1 Residual, but this time, plot the 1st 20000 points
*** and replace the "large" dot and diamond with a 1 pixel wide line.
This might help to visualize the activity.
The 3rd and 4th charts are somewhat interesting. Can you explain them a little bit.
The data presented is similar except the slopes are switched (3rd chart has a somewhat negative slope and the 4th chart has a somewhat positive slope).
Both charts appear to have four major harmonics causing four major aggregations (point densities about lines).
While charts 1 and 2 could have a single curve fitted (somewhat that of abs(sin(X)))
Charts 3 and 4 might be best served with plotting multiple lines. As to how to do this, it may require some creativity.
A guess at what to do would (iteratively)
a) filter out what appears to be stragglers from potential lines (density clusters)
b) Check for potential to curve fit multiple lines
c) if fail, tighten filter, go to a)
First pass might filter out these:
Somewhat like a "Game of Life" where a point can die, but points cannot propagate.
Jim
Jim: This is the first 20000 points as you asked. I have used the thinnest of lines. Each point represents "average" data for an 8.192 second interval which is based on 16384 FFT at 2000 Hz.
The vertical axis is tilt in degrees.
The object is an old structure made of steel with a concrete substructure. The tilt meter sits on top of the substructure 80 feet in the air.
The tilt meter reads at 200 Hz.
There are three things that can disturb the structure, thermal - traffic and construction work adjacent. If you record continuously then you have the advantage that some periods are just thermal and some traffic and thermal etc...
Whenever I do this, I am asked either what is the impact of traffic or construction. The simple answer is to code it properly and then just get all the answers and put them in a MySQL database in the cloud.
With this structure at the first meeting, a comment was made that thermal did not impact the response.
The correlation is obvious to even an old human brain.
If I plot the temperature against the tilt I get - and this shows several lines through the points
Question is do they exist or are they random - turn on lines
So the result shows the tilt is related to temp + some other factors. Now we look for the other factors.
This is traffic - thermal and construction, now it is just set theory.
Arjen: I live inside these data sets with clients asking - what does it mean.
