Tuesday, March 27, 2012
estimating design and ultimate limit of database design
how to go about it
I have allot of data - and its going to grow very rapidly in the coming
years - the core database table relates to electricity usage 15 min data. I
need to get a better understanding of what the ultimate limitations will be
on how the data is currently been managed. Current table size is 85 million
rows.
The data is saved as two tables as follows
Main Table - 18 million rows
CREATE TABLE [datData] (
[ID] [int] IDENTITY (1, 1) NOT NULL ,
[Field_ID] [int] NOT NULL ,
[Date] [datetime] NOT NULL ,
[Value] [decimal](18, 4) NOT NULL CONSTRAINT [DF_datData_Value] DEFAULT (0)
) ON [PRIMARY]
GO
With following indexes
CREATE UNIQUE CLUSTERED INDEX [IX_datData] ON
[dbo].[datData]([Field_ID], [Date]) WITH IGNORE_DUP_KEY , FILLFACTOR = 90
ON [PRIMARY]
GO
CREATE INDEX [datData0] ON [dbo].[datData]([Field_ID]) WITH FILLFACTOR =
90 ON [PRIMARY]
GO
This table is generally accessed using a simple select statement with a
WHERE clause as follows - this accounts for most of the accesses to this
table - these select statements are generally copies into temporary tables
and manipulated further
Field_ID=123 And Date BETWEEN '1-Jan-2004' AND '31-Jan-2004 23:59:59'
It is also not unusual to request max and min dates for a particular Field_ID.
At the moment everything works fine - the queries placed on the data have
not appeared to slow down - everything still appears to be operating as fast
as it was when there was only 5 million rows in the table.
I would like to get a feel for the upper limit of this table. The table
will continue to grow in both number of Field_IDs and the number of rows for
each Field ID - current there are around 5,000 field IDs - that’s an average
of 17,000 rows per field ID - or roughly 170 days of electricity data per
Field_ID.
In the future the number of field IDs could easily grow to over 100,000 and
some of these fields will contain 10 years of data. This would place the
number of rows at 35,040,000,000 or 35 billion - is this a problem?
At what point should I start to remove data from this table into an archive
- will I ever need to do this? Will there come a point when this data set
size becomes un-workable?
Anyone that can help in managing a data set of this size would be much
appreciated
matthew
Hi, Matthew
>From your description, I understand that you use the term "field" to
reffer to a place where electricity is consumed, not to a column (as
some people are used to do). Also, I understand that this table will
store exactly one row for each "field", for any given date (i.e. for
any "field", at a particular Date, there is only one Value). In this
case, I propose to use a table like this:
CREATE TABLE datData (
Field_ID int NOT NULL
CONSTRAINT [FK_datData_Fields]
FOREIGN KEY REFERENCES Fields (Field_ID),
ReadingDate smalldatetime NOT NULL ,
EnergyValue decimal (18, 4) NOT NULL
CONSTRAINT DF_datData_EnergyValue DEFAULT (0),
CONSTRAINT PK_datData PRIMARY KEY (Field_ID, ReadingDate)
)
I have changed the following:
- I change the names of the "Date" and "Value" column, because they
were too vague (moreover, "Date" is a reserved keyword); you should
change the column names to something more appropriate in your
particular case (I was only guessing with these names);
- I added a primary key, to enforce the uniqueness on
Field_ID+ReadingDate; also, this creates a clustered unique index on
these columns. For the specified queries, I think this index is the
most appropriate;
- I added a foreign key to references a "Fields" table, to enforce
referential integrity (I hope you have a "Fields" table);
- I dropped the ID column, because it seems unnecessary;
- I changed the data type of the ReadingDate column to smalldatetime
instead of datetime (because it is stored on 4 bytes instead of 8
bytes); you should do this, only if you do not need to store dates
after June 6, 2079 (I guess you don't need this) and if you do not need
accuracy under one minute (the smalldatetime datatype has an accuracy
of one minute, whereas the datetime datatype has an accuracy of about 3
milliseconds).
Regarding the number of rows in this table, I do not have any
experience with tables with billions of rows. When the performance gets
worse and if more hardware is available, you should try a distributed
partitioned view. See:
http://msdn.microsoft.com/library/en...es_06_17zr.asp
Razvan
estimating design and ultimate limit of database design
how to go about it
I have allot of data - and its going to grow very rapidly in the coming
years - the core database table relates to electricity usage 15 min data. I
need to get a better understanding of what the ultimate limitations will be
on how the data is currently been managed. Current table size is 85 million
rows.
The data is saved as two tables as follows
Main Table - 18 million rows
CREATE TABLE [datData] (
[ID] [int] IDENTITY (1, 1) NOT NULL ,
[Field_ID] [int] NOT NULL ,
[Date] [datetime] NOT NULL ,
[Value] [decimal](18, 4) NOT NULL CONSTRAINT [DF_datData_Value]
DEFAULT (0)
) ON [PRIMARY]
GO
With following indexes
CREATE UNIQUE CLUSTERED INDEX [IX_datData] ON
[dbo].[datData]([Field_ID], [Date]) WITH IGNORE_DUP_KEY ,
FILLFACTOR = 90
ON [PRIMARY]
GO
CREATE INDEX [datData0] ON [dbo].[datData]([Field_ID]) WITH
FILLFACTOR =
90 ON [PRIMARY]
GO
This table is generally accessed using a simple select statement with a
WHERE clause as follows - this accounts for most of the accesses to this
table - these select statements are generally copies into temporary tables
and manipulated further
Field_ID=123 And Date BETWEEN '1-Jan-2004' AND '31-Jan-2004 23:59:59'
It is also not unusual to request max and min dates for a particular Field_I
D.
At the moment everything works fine - the queries placed on the data have
not appeared to slow down - everything still appears to be operating as fast
as it was when there was only 5 million rows in the table.
I would like to get a feel for the upper limit of this table. The table
will continue to grow in both number of Field_IDs and the number of rows for
each Field ID - current there are around 5,000 field IDs - that’s an avera
ge
of 17,000 rows per field ID - or roughly 170 days of electricity data per
Field_ID.
In the future the number of field IDs could easily grow to over 100,000 and
some of these fields will contain 10 years of data. This would place the
number of rows at 35,040,000,000 or 35 billion - is this a problem?
At what point should I start to remove data from this table into an archive
- will I ever need to do this? Will there come a point when this data set
size becomes un-workable'
Anyone that can help in managing a data set of this size would be much
appreciated
matthewHi, Matthew
>From your description, I understand that you use the term "field" to
reffer to a place where electricity is consumed, not to a column (as
some people are used to do). Also, I understand that this table will
store exactly one row for each "field", for any given date (i.e. for
any "field", at a particular Date, there is only one Value). In this
case, I propose to use a table like this:
CREATE TABLE datData (
Field_ID int NOT NULL
CONSTRAINT [FK_datData_Fields]
FOREIGN KEY REFERENCES Fields (Field_ID),
ReadingDate smalldatetime NOT NULL ,
EnergyValue decimal (18, 4) NOT NULL
CONSTRAINT DF_datData_EnergyValue DEFAULT (0),
CONSTRAINT PK_datData PRIMARY KEY (Field_ID, ReadingDate)
)
I have changed the following:
- I change the names of the "Date" and "Value" column, because they
were too vague (moreover, "Date" is a reserved keyword); you should
change the column names to something more appropriate in your
particular case (I was only guessing with these names);
- I added a primary key, to enforce the uniqueness on
Field_ID+ReadingDate; also, this creates a clustered unique index on
these columns. For the specified queries, I think this index is the
most appropriate;
- I added a foreign key to references a "Fields" table, to enforce
referential integrity (I hope you have a "Fields" table);
- I dropped the ID column, because it seems unnecessary;
- I changed the data type of the ReadingDate column to smalldatetime
instead of datetime (because it is stored on 4 bytes instead of 8
bytes); you should do this, only if you do not need to store dates
after June 6, 2079 (I guess you don't need this) and if you do not need
accuracy under one minute (the smalldatetime datatype has an accuracy
of one minute, whereas the datetime datatype has an accuracy of about 3
milliseconds).
Regarding the number of rows in this table, I do not have any
experience with tables with billions of rows. When the performance gets
worse and if more hardware is available, you should try a distributed
partitioned view. See:
http://msdn.microsoft.com/library/e...des_06_17zr.asp
Razvan
estimating design and ultimate limit of database design
how to go about it
I have allot of data - and its going to grow very rapidly in the coming
years - the core database table relates to electricity usage 15 min data. I
need to get a better understanding of what the ultimate limitations will be
on how the data is currently been managed. Current table size is 85 million
rows.
The data is saved as two tables as follows
Main Table - 18 million rows
CREATE TABLE [datData] (
[ID] [int] IDENTITY (1, 1) NOT NULL ,
[Field_ID] [int] NOT NULL ,
[Date] [datetime] NOT NULL ,
[Value] [decimal](18, 4) NOT NULL CONSTRAINT [DF_datData_Value] DEFAULT (0)
) ON [PRIMARY]
GO
With following indexes
CREATE UNIQUE CLUSTERED INDEX [IX_datData] ON
[dbo].[datData]([Field_ID], [Date]) WITH IGNORE_DUP_KEY , FILLFACTOR = 90
ON [PRIMARY]
GO
CREATE INDEX [datData0] ON [dbo].[datData]([Field_ID]) WITH FILLFACTOR = 90 ON [PRIMARY]
GO
This table is generally accessed using a simple select statement with a
WHERE clause as follows - this accounts for most of the accesses to this
table - these select statements are generally copies into temporary tables
and manipulated further
Field_ID=123 And Date BETWEEN '1-Jan-2004' AND '31-Jan-2004 23:59:59'
It is also not unusual to request max and min dates for a particular Field_ID.
At the moment everything works fine - the queries placed on the data have
not appeared to slow down - everything still appears to be operating as fast
as it was when there was only 5 million rows in the table.
I would like to get a feel for the upper limit of this table. The table
will continue to grow in both number of Field_IDs and the number of rows for
each Field ID - current there are around 5,000 field IDs - thatâ's an average
of 17,000 rows per field ID - or roughly 170 days of electricity data per
Field_ID.
In the future the number of field IDs could easily grow to over 100,000 and
some of these fields will contain 10 years of data. This would place the
number of rows at 35,040,000,000 or 35 billion - is this a problem?
At what point should I start to remove data from this table into an archive
- will I ever need to do this? Will there come a point when this data set
size becomes un-workable'
Anyone that can help in managing a data set of this size would be much
appreciated
--
matthewHi, Matthew
>From your description, I understand that you use the term "field" to
reffer to a place where electricity is consumed, not to a column (as
some people are used to do). Also, I understand that this table will
store exactly one row for each "field", for any given date (i.e. for
any "field", at a particular Date, there is only one Value). In this
case, I propose to use a table like this:
CREATE TABLE datData (
Field_ID int NOT NULL
CONSTRAINT [FK_datData_Fields]
FOREIGN KEY REFERENCES Fields (Field_ID),
ReadingDate smalldatetime NOT NULL ,
EnergyValue decimal (18, 4) NOT NULL
CONSTRAINT DF_datData_EnergyValue DEFAULT (0),
CONSTRAINT PK_datData PRIMARY KEY (Field_ID, ReadingDate)
)
I have changed the following:
- I change the names of the "Date" and "Value" column, because they
were too vague (moreover, "Date" is a reserved keyword); you should
change the column names to something more appropriate in your
particular case (I was only guessing with these names);
- I added a primary key, to enforce the uniqueness on
Field_ID+ReadingDate; also, this creates a clustered unique index on
these columns. For the specified queries, I think this index is the
most appropriate;
- I added a foreign key to references a "Fields" table, to enforce
referential integrity (I hope you have a "Fields" table);
- I dropped the ID column, because it seems unnecessary;
- I changed the data type of the ReadingDate column to smalldatetime
instead of datetime (because it is stored on 4 bytes instead of 8
bytes); you should do this, only if you do not need to store dates
after June 6, 2079 (I guess you don't need this) and if you do not need
accuracy under one minute (the smalldatetime datatype has an accuracy
of one minute, whereas the datetime datatype has an accuracy of about 3
milliseconds).
Regarding the number of rows in this table, I do not have any
experience with tables with billions of rows. When the performance gets
worse and if more hardware is available, you should try a distributed
partitioned view. See:
http://msdn.microsoft.com/library/en-us/createdb/cm_8_des_06_17zr.asp
Razvan
Sunday, February 26, 2012
Error:The type initializer for "Microsoft.ReportDesigner.Drawing.Language" threw an except
I've been trying to get to the bottom of this error it prevents be
from loading the report in Design Mode. As of right now when opening
a report in design mode through the VS .Net 2003 IDE I get the
following:
Deserialization failed: The type initializer for
"Microsoft.ReportDesigner.Drawing.Language" threw an exception.
I've tried uninstall/reinstall, reporting services service pack 1 and
debuging the devenv.exe from another ide instance. Debug error is:
System.ComponentModel.Design.Serialization.CodeDomSerializerException:
Deserialization failed: The type initializer for
"Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
Microsoft.DataWarehouse.Serialization.XmlSerializationException:
Deserialization failed: The type initializer for
"Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
System.TypeInitializationException: The type initializer for
"Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
System.ArgumentException: Culture ID 31770 (0x7C1A) is not a supported
culture.
Parameter name: culture
at System.Globalization.CultureInfo..ctor(Int32 culture, Boolean
useUserOverride)
at System.Globalization.CultureTable.GetCultures(CultureTypes
types)
at Microsoft.ReportDesigner.Drawing.Language.GetStandardValues()
at Microsoft.ReportDesigner.Drawing.Language..cctor()
-- End of inner exception stack trace --
at Microsoft.ReportDesigner.Drawing.Language..ctor(String value)
at Microsoft.ReportDesigner.Drawing.LanguageConverter.CreateObject(String
value)
at Microsoft.ReportDesigner.Drawing.XmlStringListConverter.ConvertFrom(ITypeDescriptorContext
context, CultureInfo culture, Object value)
at System.ComponentModel.DefaultValueAttribute..ctor(Type type,
String value)
-- End of inner exception stack trace --
at Microsoft.ReportDesigner.Serialization.DesignXmlReader.ReadRoot(Type
type)
at Microsoft.ReportDesigner.Serialization.DesignXmlReader.DeserializeComponent(IDesignerSerializationManager
manager, XmlReader reader, Type root)
at Microsoft.ReportDesigner.Serialization.DesignXmlSerializer.DeserializeObject(IDesignerSerializationManager
manager, Object serializationStream)
at Microsoft.ReportDesigner.Serialization.DesignXmlSerializer.Deserialize(IDesignerSerializationManager
manager, Object serializationStream)
at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.DataWarehouseDesignerLoader.Deserialize()Hi, I got the same error here, if you already fixed this problem, please post
here waht is going on.
Thanks, Rafael
"Ricardo" wrote:
> Hi,
> I've been trying to get to the bottom of this error it prevents be
> from loading the report in Design Mode. As of right now when opening
> a report in design mode through the VS .Net 2003 IDE I get the
> following:
> Deserialization failed: The type initializer for
> "Microsoft.ReportDesigner.Drawing.Language" threw an exception.
> I've tried uninstall/reinstall, reporting services service pack 1 and
> debuging the devenv.exe from another ide instance. Debug error is:
> System.ComponentModel.Design.Serialization.CodeDomSerializerException:
> Deserialization failed: The type initializer for
> "Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
> Microsoft.DataWarehouse.Serialization.XmlSerializationException:
> Deserialization failed: The type initializer for
> "Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
> System.TypeInitializationException: The type initializer for
> "Microsoft.ReportDesigner.Drawing.Language" threw an exception. -->
> System.ArgumentException: Culture ID 31770 (0x7C1A) is not a supported
> culture.
> Parameter name: culture
> at System.Globalization.CultureInfo..ctor(Int32 culture, Boolean
> useUserOverride)
> at System.Globalization.CultureTable.GetCultures(CultureTypes
> types)
> at Microsoft.ReportDesigner.Drawing.Language.GetStandardValues()
> at Microsoft.ReportDesigner.Drawing.Language..cctor()
> -- End of inner exception stack trace --
> at Microsoft.ReportDesigner.Drawing.Language..ctor(String value)
> at Microsoft.ReportDesigner.Drawing.LanguageConverter.CreateObject(String
> value)
> at Microsoft.ReportDesigner.Drawing.XmlStringListConverter.ConvertFrom(ITypeDescriptorContext
> context, CultureInfo culture, Object value)
> at System.ComponentModel.DefaultValueAttribute..ctor(Type type,
> String value)
> -- End of inner exception stack trace --
> at Microsoft.ReportDesigner.Serialization.DesignXmlReader.ReadRoot(Type
> type)
> at Microsoft.ReportDesigner.Serialization.DesignXmlReader.DeserializeComponent(IDesignerSerializationManager
> manager, XmlReader reader, Type root)
> at Microsoft.ReportDesigner.Serialization.DesignXmlSerializer.DeserializeObject(IDesignerSerializationManager
> manager, Object serializationStream)
> at Microsoft.ReportDesigner.Serialization.DesignXmlSerializer.Deserialize(IDesignerSerializationManager
> manager, Object serializationStream)
> at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.DataWarehouseDesignerLoader.Deserialize()
>|||I have problem in Reporting Services 2005 with width of report - max. limit is probably 406,34921 cm. Is it possible to increase this limit to for exapmle 450 cm ?
From http://www.developmentnow.com/g/115_2004_11_0_0_453315/ErrorThe-type-initializer-for-Microsoft-ReportDesigner-Drawing-Language-threw-an-exception.ht
Posted via DevelopmentNow.com Group
http://www.developmentnow.com
error:no report server found on the specified machine while setting up reporting services config
Hello,
I have installed sql server 2005 along with reporting services... though i am able to design report using business intelligence studio... i am unable to access the report server.... while trying to start the reporting services configuration manager it says no report server found on the specified machine...Invalid Namespace... the installation is local ...
Due to this problem inspite of designing the entire report i am unable to deploy it on the web..since it is asking for a report server...
Can somebody please help me on this...
Thanks in advance...
Nirupa
Hello Nirupa,
Can you make sure you have IIS installed and running? If so, make sure you have /Reports and /ReportServer showing in IIS management.
Then, check to see if you have the following three folders in the C:\Program Files\Microsoft SQL Server\MSSQL.x\Reporting Services directory (where x is the component number for RS). LogFiles, ReportManager, & ReportServer
Hope this helps.
Jarret
|||Hi Jarret,
Thanks for your suggestions first,
yes I installed IIS and it is running,
and
I couldn't find any folder C:\Program Files\Microsoft SQL Server\MSSQL.1\ Reporting Services directory
...LogFiles, ReportManager, & ReportServer
I could see only path C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\ Backup
'' \Bin
'' \Install
'' \LOG
'' \repldata
'' \TemplateData
can you suggest me what might had happen.... or what shall be done
Thanks
Nirupa
|||Do you have a MSSQL.2, MSSQL.3, or MSSQL.4 folder? If so, look in each for the 3 folders I listed in my last post. If you don't have these folders, you probably should reinstall Reporting Services.
Jarret
|||Hello Jarret
I do not have the folders that you have mentioned....As you have mentioned will probably need reinstallation
Thank you
Nirupa
|||That is correct, if you don't have these folders, you don't have RS installed. Try to re-install and see if that fixes your problem.
Jarret
|||Hi Jarret,
Thanks for your suggestions.As you mentioned, I reinstalled SQL reporting services and got the above mention folders(MSSQL.1,MSSQL.2,MSSQL.3) and also I successfully configured reporting services.While trying to deploy my reporting services application in to localhost , I got the below mentioned error : " The attempt to connect to the report server failed. Check your connection information and that the report server is a compatible version. (Microsoft.ReportingServices.Designer) "
can you please sugguest me work around for this issue.
Friday, February 17, 2012
Error: Subreport could not be shown.
a single query, since on the design window I can see only one page, my
initial solution was to create 4 reports, and they work fine, but the
end user needs to see the 4 forms at once to send it to a pdf.
Now, I created a new report and add the 4 reports as a subreports, but
I am getting the message "Error: Subreport could not be shown. "
I will apreciate any help.
Thank you
GustavoHi,
Just check the right report you have selected from the drop down for sub
reportand most importantly all the parameters are selected propely. I doubt
it should be some missing parameters.
Amarnath.
"gvt99@.hotmail.com" wrote:
> I have to print 4 forms one form per page from a datatable generated by
> a single query, since on the design window I can see only one page, my
> initial solution was to create 4 reports, and they work fine, but the
> end user needs to see the 4 forms at once to send it to a pdf.
> Now, I created a new report and add the 4 reports as a subreports, but
> I am getting the message "Error: Subreport could not be shown. "
> I will apreciate any help.
> Thank you
> Gustavo
>|||Gustavo,
You are probably missing some parameters. You must supply the
subreports with parameters from the Master report.
Also, Amarnath, dont forget that if the report has temp tables, it will
not automatically fill those parameters.
regards,
Stas K.