You can always disable the like/dislike bar for a while. In the end dislikes still count towards your viewer engagement the same the likes do so it will help out your video ranking rather than be bad for it. Although I know it can be disheartening especially if you have a small channel...