Public sector programs usually have higher administrative costs — all relevant costs considered — than corresponding private sector programs. The public sector program is funded by taxation. That means the public sector doesn't have to worry so much about marketing or meeting payroll on commercial revenue alone. That will bring significant cost savings on administrative matters. But you can't stop counting there.
The deadweight loss from taxation is perhaps twenty percent or more. (It depends on which tax you consider as "the marginal tax" and there is not a simple factual answer to that question.) That should be factored into any comparison, even if you define that cost as "not an administrative cost."
The public sector also engages in less monitoring of who receives its services. If you're 67 and have worked a lifetime in this country, usually you can receive Medicare benefits. The "indiscriminate" nature of the program may be either a net social cost or a net social benefit but certainly it should not be counted as zero or ignored.
If you favor "indiscriminate" programs over targeted programs, OK. But the accompanying lesson is not one about the relative efficiency of the public sector at a comparable task. The lesson is that sometimes the public sector can be more effective when you don't wish to discriminate in supplying a particular kind of service.