It means if they were not to listen to him, let them remember that he was not responsible for the eventuality that they may be seized by some punishment.

The Holy Prophet ﷺ said about Sayyidna Shu'aib that he was an orator among prophets. His eloquence was exceptional. He went to its farthest limits in order to make his people understand his message and take to the right path. But, even after having listened to him, the response of his people was different from that given by ignorant diehards to their reformers. They threw taunts at him, made fun of him, and said:

أَصَلَاتُكَ تَأْمُرُ‌كَ أَن نَّتْرُ‌كَ مَا يَعْبُدُ آبَاؤُنَا أَوْ أَن نَّفْعَلَ فِي أَمْوَالِنَا مَا نَشَاءُ ۖ إِنَّكَ لَأَنتَ الْحَلِيمُ الرَّ‌شِيدُ

0 Shu'aib, does your salah (prayer) command you that we should give up what our fathers used to worship or that we give up doing with our wealth whatever we wish? You are provenly the man of wisdom and guidance - 87.

The salah (prayer) of Sayyidna Shu'aib (علیہ السلام) was well known among his people as he was constantly devoted to it. Therefore, they referred to it tauntingly in the sense that (God forbid) his salah taught him to say all those 'wild' things. From what they said we learn that they too thought that religion is restricted to acts of worship, and that it had nothing to do with material dealings. Let everyone earn and spend one's wealth at will. Placing any restriction on that is no job of religion - as many people, even in this time of ours, generally believe, in total ignorance of the real fact.