Filter for magnetometer

[devxpy]

I am trying to do some distance calculations for this project using the magnetometer.

But i always get a stale magnetic interference when the sensor is stationary.
I have tried several moving average algorithms. They improved the overall signal quite a bit.

But its still nowhere near what the project requires.

I have tried several magnets. Neodymium, speaker magnets, varying from little to high intensity and i am getting similar results..

Can you advise me on how can I implement a filter?

[peterhinch]

If you're using an STM processor you might like to look at my FIR filter library. This uses the inline assembler for processing real time data from an ADC so is ridiculously over-specified for data read from the magnetometer. The algorithm for an FIR filter is similar to a moving average - you just have multiplication by a coefficient at each sage. So you could easily implement it in Python if you're using another platform.

Specifying the filter might be easier if you gathered some data over a period and did a DFT to view the results in the frequency domain.

Before considering filtering I'd try to figure out where the interference is coming from. Do you have any local source of alternating magnetic fields?

[devxpy]

Thanks!
I am using a wemos D1 mini.

I will try the FIR filter you suggest.
Is applying a filter before this module does its thing really beneficial?
I do all the heavy lifting on my main pc that receivies the data from this module over UDP so performance is not really an issue.
(I would apply the filter in numpy which would be real easy to implement for me)

I am already gathering the data for a few seconds to figure out the min-max normalization parameters. So should be easy to gather the DFT data. I will do it right away.

I suspect it could be from the Wemos D1 mini. I have it a little close to the sensor.
Here is a picture

[devxpy]

After FFT
Before FFT ( sample S. No. on the x axis, EDIT: processed data on the y axis)

EDIT: Here is the function that I use to process that data right after mag.xyz

def process(self, data):
    # get max
    data = np.max(np.abs(data))
    # linearlize
    np.power(data, -3)

For data collected for 5 secs while moving the magnet continuously

[peterhinch]

That was quick! To see the noise spectrum you need to measure with the magnet removed or at a fixed position. It would be good to know the units on the frequency scale of the FFT.

[devxpy]

I am not sure which units you are talking about.
We need the y/x axis before/after the fft?

I have updated previous comment to be more relevant

Here is the code i used to create the FFT

import numpy as np
import matplotlib.pyplot as plt

y = np.abs(np.fft.fft(data))
x = np.fft.fftfreq(len(y), 5)

fig, ax = plt.subplots()
ax.plot(x, y)
plt.show()

Following information is for a magnet kept at constant distance (~10cm) from the sensor
Before FFT
After FFT

Here is the "data"

Following information is for no magnet near the sensor
Before FFT
After FFT

Here is the "data"

I have zoomed in the graphs a bit because they look like this otherwise

[peterhinch]

I was hoping to see frequency values in Hz (or milliHz) on the X axis of the FFT graphs. This would give us some idea of the best cutoff frequency of any low pass filter - or whether low pass filtering is feasible. Clearly you want to pass frequencies below a certain level (2Hz?) as the user might move their hand below that rate. Hopefully the bulk of the noise energy is above that frequency, otherwise any filtering is doomed. Filtering can only work if the noise is outside the band you need to pass.

[devxpy]

That made it very clear. Thanks for the explanation. I will try some stuff out and get back to you :)

[devxpy]

Btw, Did you see that image of my circuit? Do you think that could be causing interference?

[devxpy]

These should be in Hz.

No magnet
Moving magnet

I feel like doing a diff of these would help?

here is the modified code

import numpy as np
import matplotlib.pyplot as plt

y = np.abs(np.fft.fft(data))
x = np.fft.fftfreq(len(y), 5 / len(y))

fig, ax = plt.subplots()
ax.plot(x, y)
plt.show()

[devxpy]

Here is a diff

A low pass filter for 10 Hz seems reasonable.

[peterhinch]

10Hz is certainly worth a try.

I wondered about alternative technologies for your digital theremin which strikes me as a cool project :)

Sensing magnets over distance is quite tricky and very nonlinear. I've thought of two other options. One is to use OpenMV which can track brightly coloured objects. The second, if it's acceptable for the glove to carry electronics, is to use a battery powered ESP32 with your IMU. Perform sensor fusion and pass heading, pitch and yaw data to the instrument via WiFi. You would then have three degrees of freedom for volume, pitch and, well, whatever.

[devxpy]

Full disclaimer, this is a collab between me and hasan
If it's okay, can we continue our discussion in private?

[peterhinch]

I can be contacted via PM on the forum. Username pythoncoder.

[devxpy]

Looks like i'm not eligible to do PMs right now so never-mind

Here are some images of what i've created on the hardware side until now
I will have a working demo video with the pads working in some time.

We are targeting a self contained device, just like you suggest, that has a battery and an esp on board, rather than a CV system which is rather common and boring.

• Digital theremin is a good analogy, but we were targeting a more complete instrument sort of thing..
  Using the magnetometer to calculate the distance between two hands, that would simulate a real fretboard
• We are also planning to do another version that does something like this using accelerometer.

The linearity is pretty good for distances that are not too near the magnet. I do see some weird behavior for distances very near to the magnet but its fine otherwise.
Not perfect but workable

You would then have three degrees of freedom for volume, pitch and, well, whatever.

That sounds very interesting. I will be sure to try this out.

P.S., i saw some good results with the 10Hz filter. the signal is quite stable now while stationary.
( I had a plan to factor in the accelerometer readings and check if the hand is stationary and cut-off the sounds but low-pass filter is definitely a more elegant solution )

Thanks for all your guidance

[peterhinch]

I'd assumed that the magnet was on the glove and the electronics were static. Given that the electronics are on the glove there are many possibilities.

It occurred to me that the sensor fusion idea could be taken further to provide more degrees of freedom. But it would need a bit of maths. With a mathematically stable platform provided by sensor fusion you could derive position by double integrating the acceleration with respect to that platform. You'd have to take G out of the processing for the Z axis but you'd gain three positional DOF. OK, double integration is potentially error-prone but you have a feedback loop provided by the fact that the outcome is audible. I've never actually tried to perform such inertial measurements but it would be fun to try. I think you'd need an ESP32 because of the sheer amount of code in the libraries.

What do the wires to the fingers do?

[devxpy]

Double integration! That never occurred to me. I could do sensor fusion on my main pc where I do the calculations anyways?

The fingers are for playing notes, like on a violin fretboard. So instead of strings, you press these pads

P.S., i applied Savitzky–Golay filter and this is the level of smoothness i got.

[devxpy]

Here is that demo video

[peterhinch]

Neat! That Savitzky-Golay filter is new to me although I'm familiar with the underlying concepts. I'll study that further. It certainly produces impressive results.

As to which processing is done locally and which on the PC it depends on data rates and WiFi latency. I have my doubts about doing sensor fusion and double integration on the PC because of these potential lags.

As we have the library for the MPU9250 which is proven with the fusion library doing it onboard should be easy if RAM is adequate. The vector rotation and double integration shouldn't add too much code. In that way any WiFi latency will merely result in a delay in playing the music rather than compromising the accuracy of the calculations. My 'finger in the air' feeling is that for double integration to stand a chance of working the data needs to be timely.

But that's a 'hand waving' guess rather than a provable idea.

[devxpy]

Alright, will put your suggested sensor fusion module on my board and take it for a test drive.

I was merely suggesting to do on PC due to numpy / scipy.
But you do make a good point, I will try to do it on the board itself.

[peterhinch]

From an electrical engineer's perspective that Savitzky–Golay filter is similar to a finite impulse response (FIR) filter. Both involve convolution with a set of predefined coefficients.

You may have seen my fast solution for the Pyboard and the case where the X axis represents time.

[devxpy]

I didn't have to choose any coefficients manually since the Savitzky–Golay function in scipy does it automatically based on 2-3 simple parameters. That made it really easy and fast for me to implement

[peterhinch]

Curiosity got the better of me and I did some tests to see if deriving position by double-integration was feasible. I did the tests on a flat surface so that the sensor fusion and vector rotation was avoided. This also avoided the complexity of factoring in gravity by moving only in the horizontal plane. So this was a best-case scenario in terms of potential error accumulation.

My conclusion is that it's a non-starter. Mathematical purity is trounced by the engineering reality: the signal is dwarfed by the random noise from the sensor. It is possible that a sophisticated filter algorithm might overcome this, but I have my doubts.

[devxpy]

That's awesome. Random noise will probably be fixed by the savitzky filter. I had tons of it earlier as well.

I apologise for being unresponsive for the past few days. We're short on funds and working on a different project to raise some for a while.
http://exam.pinnacleiitjee.com/quiz/

[devxpy]

Please provide me your bitbucket username /email. I will add you to our repo which has the code for communication between esp and pc, and also the savitzky filter. It's implemented using buffers that try to do the processing in almost real-time.

[peterhinch]

YHM.