SQL Server Source | More than 50 calls to sp_spaceused

[tolemac]

I have been testing airbyte using MS SQL Server as source and I can see in the logs more than 50 calls to funcion sp_spaceused per each table.

That add to many noise to the logs, to have continuous repeating message as that:

...
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(getTableEstimate):242 Querying for table estimate size: EXEC sp_spaceused N'"admin"."FAR_FACTURAECAB"'
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(lambda$getTableSizeInfoForStreams$6):158 Stream "FAR_FACTURAECAB"."admin" size estimate is 3915776, average row size estimate is 78
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(getTableEstimate):242 Querying for table estimate size: EXEC sp_spaceused N'"admin"."FAR_FARACT"'
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(lambda$getTableSizeInfoForStreams$6):158 Stream "FAR_FARACT"."admin" size estimate is 8192, average row size estimate is 49
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(getTableEstimate):242 Querying for table estimate size: EXEC sp_spaceused N'"admin"."FAR_FLUJO"'
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(lambda$getTableSizeInfoForStreams$6):158 Stream "FAR_FLUJO"."admin" size estimate is 8192, average row size estimate is 2730
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(getTableEstimate):242 Querying for table estimate size: EXEC sp_spaceused N'"admin"."FAR_GASTO"'
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(lambda$getTableSizeInfoForStreams$6):158 Stream "FAR_GASTO"."admin" size estimate is 112001024, average row size estimate is 317
2024-06-07 12:11:55 source > INFO main i.a.i.s.m.MssqlQueryUtils(getTableEstimate):242 Querying for table estimate size: EXEC sp_spaceused N'"admin"."FAR_HISTORIA"'
...

Is that the normal situation?

What is the reason for so many calls?

[b4stien]

Did you find any answer for that? In a big connection (440 streams) it's been looping for more than 30 minutes on those calls on my end.

[b4stien]

Found it, it's a quadratic call to this function, I've opened #67291

[tolemac]

No, I have abandoned this option. I'm doing data movement with PanSynchro and we're super happy. Airbyte didn't give us the agility we needed.