<p>I bought a self-made Reprap Prusa Mendel 3 printer, modified to be built from cheaper materials, and immediately after the deal I got various problems. I fixed the majority of them, but don't know what the reason of the strange extruder behavior is: The stepper motor is not rotating while extruding filament, it's moving back and forth with small steps instead. I made some footage, <a href="https://www.youtube.com/watch?v=OcF9qxtxIO0" rel="nofollow noreferrer">Reprap Prusa Mendel 3d printer extruder problem</a>, hoping that someone has met a similar problem before.</p>

<p>If so, please tell me what to do to make it work as it should.</p>

<p>The obvious option while encountering this problem was to slacken the bolt which holds this parts together (in the left bottom corner of the video), but that did not help.</p>

<p>Any ideas are very welcomed. Thanks for your time. I hope this is an appropriate kind of question here.</p>

Your controller board probably requires calibration.
It sounds like, maybe, the extruder's stepper motor is
not receiving sufficient
current, to make it turn.
Or
, somewhat confusingly, maybe the stepper is
receiving too much current
, and overheating.
You don't say which controller board you are using, but regardless, there should be an adjustable potentiometer on the board, next to each of the stepper drivers, or on the stepper driver daughter boards.  Like so,
This potentiomenter adjusts the
reference voltage
used to control the stepper motor. From this
reference voltage
, and the resistance of the stepper coils, one can determine the current, which is used to drive the stepper motor.
For the stepper driver of the extruder, you could try turning this adjustable potentiometer slightly, in order to provide more current to the stepper, in turn to provide
sufficient torque
such that the motor is able to turn.
Or
, less current to stop the stepper from overheating.
The adjustments
can
be made whilst the power is on, but a
non-ferrous
(i..e. plastic) screw driver should be used, so as to avoid short circuits. Also care needs to be taken, when turning the potentiometer, as they have been known to just fall apart whilst being turned. If you are paranoid, then make micro adjustments with the power turned off, and then turn back on to check the behaviour.
Note
: it should go without saying that one should
never disconnect a stepper whilst the power is on
, as both the driver and the stepper motor may be irrevocably damaged.
The photo above is taken from
POTs Calibration – RAMPS 1.4
.
If a POT is set too high then the associated stepper driver will tend to overheat and go into over-temperature thermal shutdown (to prevent damage to its components). The first sign of overheating is erratic stepper motor behavior. Typically, this can be recognized by the sounds of the stepper motor suddenly losing power (thermal shutdown). If no load or movement is required of the motor, it is hard to detect whether it is over-powered as the driver is barely producing any heat.
and
Conversely, if the POT is set too low, the stepper motor can enter an underpowered state. This can be recognized by a lack of holding torque and a stepper motor that is skipping steps because the necessary movement  requires a higher power demand than the POT setting allows for.
Driver cooling
In addition to the possibility of the stepper motor over heating, it could be possible that the stepper driver is overheating, although the symptoms may be different, to those that you are experiencing. Regardless, you may still find it advantageous to cooler the controller/driver board with a fan that is always on (not temperature controlled).
Additional reading
RigidWiki -
Stepper Driver Adjustment
, which goes into further detail about the adjustment of the potentiometers, that I outlined above, as well as the
reference voltage
and the adjustment thereof.
RepRap Wiki -
RepRapPro Setting Motor Currents
describes a different controller to yours, but goes into the process of adjustment, and description of the
reference voltage
(which is applicable to all boards):
The wiper on each potentiometer generates a DC voltage that is sent to the chip. This is the reference voltage; it defines how much current the stepping motor driver chip supplies to the motor. The bigger the reference voltage (VREF), the higher the current (A) that the chip will send to the motor. For most NEMA14 motors, the current maximum is 1A, but this will generally cause it to get warm, so a setting of 750mA is recommended. For NEMA17 motors, depending on size, the limit on current is generally between 1.3A and 1.7A. If you drive stepper motors with more current than they were designed for, the motor will get hot, and may be damaged.
Pololu -
A4988 Stepper Motor Driver Carrier with Voltage Regulators
- this is a very common stepper driver.
MyHomeFab -
DRV8825 Adjust Stepper Current
goes into the adjustment of the reference voltage, for the commonly used DRV8825, which is an alternative to the popular A4988.
This thread, about non-actuating steppers, may also be useful,
Motors
, which mentions
setting the trimpots
and points the OP to RepRap Wiki -
Pololu stepper driver board
, which, in turn, refers to this thread,
Strange stepper behavior
and this video,
video-2012-02-02-16-37-26.mp4
, which describes a jitter in the stepper behaviour.