Rhythm or notes: what is more important?
I'll answer this question first from the point of view of a professional sight-reader. If you're accompanying an opera rehearsal, or playing chamber music, both are equally important. If you play the right rhythms, but the wrong melody and harmonies, the piece will be unrecognisable. Ditto if the melody is correct, but the rhythm is random.
But that's not all. Let's say you have a passage marked pp, vivace and all in staccato. If you play every single note right, with the correct rhythms, but you play ff, lento and with lots of pedal, it will be a disaster.
So, in order to be a good pro sight-reader, do you need to get everything exactly right straightaway? No! You scan the music, get a good idea of how it should sound, and concentrate on the essentials. You may leave out a load of notes, or get some details of the rhythm wrong, but you produce something recognisable, you stay together with your colleagues and you make music.
That's the goal to aim for in advanced sight-reading, but if you're near the beginning of your sight-reading journey, it will be too much for you to try to assimilate everything at once. Quite possibly your brain will freeze up, and you will achieve nothing.
What should you do? Work on each element separately first:
If you want to work on rhythm, just clap or tap the rhythms.
If you want to work on harmony, find a piece with a lot of chords and enjoy the progression of one chord to the next. To hell with the rhythm!
If your left hand is weak in sight-reading, read just the left hand part. You can also find simple pieces for left hand alone, for example here.
And so on...
Give it time. As you become more fluent in each element, it will become easier to put them together.
Finally, a big bravo for anybody who has the courage to record their sight-reading and post it for all to see and hear. You have my undying admiration!